分析の問題のトラブルシューティング

このページでは、アプリケーション分析の特定のシナリオにおける展開の問題と回避策に関する一般的なトラブルシューティング手法について説明します。

General Troubleshooting Issues

If you encounter problems with your Analytics deployment, check the logs for errors and warnings.

These logs are useful to:

  • Determine if the Analytics Dynamic Service is enabled or disabled.
  • Determine if the Analytics Dynamic Service encounters errors sending messages to the Analytics Agent. For example, when the Analytics Dynamic Service is not able to communicate with the Analytics Agent due to invalid connection configuration.
  • Determine if messages are being dropped by the Analytics Dynamic Service because its internal buffers are full.
  • View the configurations used on startup.

The following components write log information.

  • Analytics Agent: The Analytics Agent writes log messages to files in the <analytics_agent_home>/logs directory.
  • Events Service: The Events Service writes log messages to files in the <events_service_home>/logs directory. The events-service-api-store.log file can help with troubleshooting.
  • Java Agent: The Analytics Dynamic Service is built into the Java Agent and writes logs to the same file as the app agent. The logs are in the <application_home>/<app_agent_home>/logs directory.
    • The primary log file to use for troubleshooting is the file named agent.<timestamp>.log . Search the file for messages written by the Analytics Dynamic Service.
  • .NET Agent: The .NET Agent writes log messages to files in the %ProgramData%\AppDynamics\DotNetAgent\Logs directory. By default, the .NET Agent writes Analytics logs from the com.appdynamics.ee.service.analytics.Analytics logger to determine if the Analytics Dynamic Service is enabled/disabled and to view the configuration on startup. Errors are logged in the warnfile .
    • The com.appdynamics.CONFIG.AnalyticsDynamicServiceConfigListener logger gathers Analytics Dynamic Service configuration details from the Controller. Ensure that the eventServiceURL comes from the Controller.
    • The com.appdynamics.analytics.EventsServiceSink logger provides information about the communication between the .NET Agent and the Events Service.
    For additional logger information, see .NET Agent on Windows Logging.

Configuration Issues

Verify that your configuration settings are properly configured with the required account name and key. Slashes in account names and key values must be escaped.

Windows Commands

On Windows, you cannot delete a log file with the del command when the Analytics Agent is collecting log data from the file.

Do not use robocopy commands to move files on Windows. Splunk AppDynamics recommends that you use the move command instead.

Starting the Analytics Agent Issues

If you are running the Analytics Agent as an extension to the Machine Agent with JRE 1.8.0_171 or later, starting the Analytics Agent with encrypted credentials will fail. Workarounds include:

  • Disable encryption in the Machine Agent
  • Use a standalone Analytics Agent with encryption
  • Use JRE 1.8.0_162 to run the Machine Agent with encryption

PID File

If an instance of the Analytics Agent terminates and leaves behind a process ID file (PID file), the next agent startup will fail with the following error:

java.lang.RuntimeException: Unable to create file [D:\AppDynamics\analytics-agent\analytics-agent.id] to store the process id because it already exists. Please stop any currently running process and delete the process id file

You can use the -f flag option while starting the agent. This option causes a pre-existing process id file to be deleted. The flag is not required when you are starting the agent as a Windows service.

Use the -f flag as follows:

  • UNIX type OS: start -f
  • Windows CLI: start -f
注:

The -f flag option only works for the standalone Analytics Agent. This option does not work for the Analytics Agent embedded in the Machine Agent. In versions 20.6 and later of the Machine Agent, the bundled Analytics Agent no longer drops a PID file.

For the Analytics Agent running inside the Machine Agent (embedded mode), upgrade the Machine Agent to 20.6. You no longer need to delete PID files from previous runs where the agent crashed.

High CPU Usage in Transaction Analytics without Analytics Agent (Agentless Analytics)

Reaching the Transaction Analytics license limit may cause high CPU usage in older Java Agents. Splunk AppDynamics recommends that you upgrade to the Java Agent version 20.3 and later. Look out for high CPU and memory usage in the .NET Agent.

Analytics Data Issues

Clock Management

Splunk AppDynamics recommends maintaining clock-time consistency throughout your monitored environment. If Analytics metrics are consistently reporting zero, confirm that clocks are synchronized across the application, Controller, and Events Service nodes.

Timestamps

There are potentially four time zones involved with Log Analytics:

  1. The timestamp and time zone from the log file.
  2. The event timestamp and pickup timestamp time zones can be different from those in the log for a number of reasons:
    1. When the time zone is overridden.
    2. The time zone is not provided correctly in the log.
    3. The timestamp and time zone parsing goes awry.
    4. When no time zone is specified in the log timestamp, then local time is assumed.
  3. The Events Service time zone and the Events Service stores all timestamps in UTC time.
  4. The browser used to view the Analytics data in the Controller UI (such as the Event Timestamp column in the UI search results or the Time Picker widget) converts all timestamps to the browser's local time.

Environment Variables

If you are not seeing or only seeing partial Analytics data in the Controller, configure the following environment variables.

Environment VariableDescription
appdynamics.analytics.agent.send.attempt.max The maximum number of times that the Dynamic Service running in the agent will try sending the Analytics data after a communication failure.
appdynamics.analytics.agent.send.attempt.pauseMillis The wait time between attempts to send Analytics data to the Analytics Agent after a communication failure.
appdynamics.analytics.numOfSinkWriterTasks The number of sinks used by the Dynamic Service running in agent to send data to Analytics Agent. The default number is 2 and can be increased as needed.

Log Analytics Missing Field Extractions Issues

If you are missing fields in your Log Analytics data that you expect to see based on your source rule configuration or using regex (including grok patterns) in your field extraction, you may encounter a performance safeguard.

If a regex pattern takes more than five seconds to match against a log line, the attempt to extract the fields is terminated. No further processing that requires the extracted fields will occur. As a result, some fields may be missing when viewed on the Controller for that log line. In this case, the following error message appears in the Analytics Agent log:

[ERROR ] java.util.concurrent.TimeoutException: The current regex has spent 5 seconds attempting to match the log line, further processing has been stopped for this log line.

Another reason for missing fields is if the log line doesn't contain the field to be extracted as defined in the pattern.

Custom Analytics Metrics Issues

If you are having issues with metrics created from saved searches or with the alerts performance for those metrics, try increasing the query batch size. You can increase the size using the analytics.scheduledqueries.batch.size Controller setting in Controller Settings > Controller Configurations. The default value for this setting is 5.

See 2023-10-19_05-04-05_Access the Administration Console for information about accessing the setting.

Business Transaction Events Limits

The Analytics Dynamic Service sends messages to the Events Service where the request body is an array of event segments. A business transaction event consists of one or more segments that are related to each other by the business transaction requestGUID. There are ingestion limits related to messages.

  • Event (segment) size: The maximum size of an individual business transaction segment collected by the Analytics Dynamic Service is 0.1 MB. This limit is defined by the appdynamics.analytics.message.maxSizeBytes Java system property. To change this value, pass it on the command line as a system property when the Java Agent starts. For example:

    '-Dappdynamics.analytics.message.maxSizeBytes=1024000'
  • Events per request: The maximum number of segments per request is defined by the appdynamics.analytics.agent.send.batch.items.max Java system property. The default value of this property is 16. To change this value, pass it on the command line as a system property when the Java Agent is started.

  • Message Size: This limit refers to the size of a single request body sent to the Events Service, which is usually an array of event segments. Publish requests for all event types are limited to 1 MB. If the limit is exceeded, you will see exceptions in agent log file and messages in Events Service logs.

Agent-Side Metrics

Analytics components in the Java Agent reports performance metrics. These metrics can be used to monitor the health of Transaction Analytics reporting in the agent/node. These metrics are under the Agent|Analytics|<Metric Name> path in the Metric Browser.

Metric NameMetric Time Rollup TypeMetric Cluster Rollup TypeMetric Description
Collector Objects CountCurrentIndividualThe current count of Analytics data collectors in use.
Messages SentSumCollectiveThe number of messages sent.
Messages Dropped Since Input Queue FullSumCollective

The number of dropped messages due to the input queue being full.

Messages Dropped Since Size ExceededSumCollective

The number of dropped messages due to message size exceeding limit.

Message Send ErrorsSumCollectiveThe number of transient and permanent errors in event registration or publish.

Message Latency (ms)

AverageIndividualLatency calculated in milliseconds.
Pending Messages In QueueCurrentIndividualMessages pending in the buffer.
Sink Writers CountCurrentIndividualNumber of active sink writer threads.

ファイアウォールの考慮事項

Analytics エージェントを使用しないトランザクション分析では、Java/.NET エージェントとイベントサービス間の接続が必要です。Java/.NET エージェントから SaaS またはオンプレミスのイベントサービスへのリクエストをブロックするファイアウォールがある場合は、分析データのずれを回避するためにファイアウォールを開く必要があります。「エージェントとコントローラの接続」を参照してください。エージェントとコントローラの接続

SaaS ベースのインストールでは、(「エージェント側コンポーネントのインストール」で説明されているように) \conf\analytics-agent.properties ファイルで http.event.endpoint 設定を変更することで分析エンドポイントを設定します。たとえば、http.event.endpoint=http://analytics.api.appdynamics.com:443 のようになります。

ファイアウォールルールにより、ホスト名ではなく特定の IP アドレスを使用する必要がある場合は、次の情報に注意してください。(ファイアウォールルールを設定した後でも)収集されたトランザクション分析データを予定どおりに表示できず、ログに次のような Connection Reset メッセージが繰り返し表示される場合は、ファイアウォールルールに正しい IP アドレスが含まれていない可能性があります。

[2015-12-18T16:08:39,907-06:00] [INFO ] [AD Thread Pool-Global0] [o.a.http.impl.execchain.RetryExec] I/O exception (java.net.SocketException) caught when processing request to {s}->https://analytics.api.appdynamics.com:443: Connection reset

ファイアウォールルールに正しい IP アドレスが含まれていない可能性があります。

SaaS 環境では、analytics.api.appdynamics.comfra-ana-api.saas.appdynamics.com および syd-ana-api.saas.appdynamics.com がラウンドロビン DNS エイリアスであり、次の例のように複数の DNS(54. と 52.)に解決される場合があります。

Name: analytics.api.appdynamics.com Address: 54.213.173.141 Name: analytics.api.appdynamics.com Address: 52.88.111.157
Name: fra-p-con-2.saas.appdynamics.com Address: 52.28.42.67 Name: fra-p-con-2.saas.appdynamics.com Address: 52.57.96.225
Name: syd-p-con-1.saas.appdynamics.com Address: 54.153.248.179 Name: syd-p-con-1.saas.appdynamics.com Address: 54.153.246.219>

Amazon Web Services(AWS)は、使用される IP を制御します。そのため、それらの IP は随時変更される可能性があります。AWS は現在の IP アドレス範囲を JSON 形式でパブリッシュするため、ホスト名に対してファイアウォールを開くことができない場合は、AWS IP アドレス範囲をダウンロードできます。AWS IP アドレス範囲が変更されるたびに通知を受けるようにするには、Amazon SNS を使用して通知を受信するように登録します。

SaaS 分析環境の AWS リージョンについては、「分析 IP の範囲No Content found for /db/organizations/splunk/repositories/appdsaas-25_10_0/content/documents/AppDynamics/common/saas_paa/c_saas_migration_saas_domains_and_ip_ranges.dita#SaaS-Domains-and-IP-Ranges_485261539」を参照してください。リストされている各リージョンの AWS IP 範囲を確認できます。