Troubleshoot data ingestion for AI Agent Monitoring

Troubleshoot issues with the LLM Providers data integration and trace ingestion.

Attention:

Alpha features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this alpha feature available in its sole discretion and may discontinue it at any time. These documents are not yet publicly available and we ask that you keep such information confidential. Use of alpha features is subject to the Splunk Pre-Release Agreement for Hosted Services.

The following documentation links apply to users who want to store conversation data in Splunk Observability Cloud. Use the following links to navigate between the pages in this release:
The following topics describe how to troubleshoot the following data ingestion issues:

Prerequisite

The troubleshooting topics on this page assume that you have completed the steps in Set up AI Agent Monitoring.

LLM Providers data integration can’t be saved

Saving the configuration for the LLM Providers data integration fails with the following error:
CODE 
Splunk Observability Cloud could not establish a connection with LLM provider. Review your authentication credentials and try again.
  1. Use the Splunk Observability Cloud main menu to navigate to Data Management > Deployed integrations. In the LLM Providers data integration:
    1. Verify that the Base URI url field isn't missing trailing / symbols. Review the following example URLs for common providers:
      LLM provider Example URL Example models
      OpenAI https://api.openai.com/v1/

      • gpt-4o

      • gpt-4o-mini
      Google Gemini https://generativelanguage.googleapis.com/v1/

      • gemini-2.5-pro

      • gemini-2.5-flash-lite
      Azure OpenAI

      https://<YOUR_AZURE_BASE_URL>/openai/v1/

      Replace <YOUR_AZURE_BASE_URL> with your Azure URL.

      • gpt-4o

      • gpt-4o-mini
      AWS Bedrock

      https://bedrock-mantle.us-west-2.api.aws/v1/

      Note: Your URL may vary depending on your region.

      • openai.gpt-oss-120b

      • mistral.mistral-large-3
    2. Verify that the API token you provided is active.
  2. Resolve the following potential issues with your model or instrumentation:
    Issue Description Solution
    Model is too large The model is too large, which causes evaluations to be generated too slowly and may result in timeout errors. Evaluations may incur higher costs in this scenario. Change to a smaller model.
    Model is too small A small model can yield inaccurate evaluations. Change to a larger model.
    Incompatible model An incompatible model can yield an incompatible output, which causes errors. Change to a compatible model.
    Instrumentation or telemetry issues Misconfigurations can prevent requests from being accepted or cause incompatible results. Troubleshoot your instrumentation to ensure that you're using the correct headers and connection.

Trace data is no longer being ingested

The LLM Providers data integration was correctly configured. You confirmed that trace data was being ingested because the APM > AI trace data page displayed traces with a value in the Quality column. However, the AI trace data page is no longer showing new trace data.

To resolve this issue, use the Splunk Observability Cloud main menu to navigate to Data Management > Deployed integrations. Proceed with the following steps to troubleshoot the LLM Providers data integration.

  1. Verify that the rate limit isn't exceeded and you haven't run out of tokens for your LLM provider.
  2. Verify that the LLM model provided in the Model name field is still supported by the provider.
  3. Verify that the API token you provided is active. Rotate the token if needed.
  4. Verify that the Number of evaluations you set aligns with your environment. Splunk Observability Cloud stops running evaluations once the number of evaluations per minute is reached.
  5. Verify that the Sampling rate isn't set to 0.