Troubleshoot data ingestion for AI Agent Monitoring

Troubleshoot issues with the LLM Providers data integration and trace ingestion.

The following topics describe how to troubleshoot the following data ingestion issues:

Prerequisite

The troubleshooting topics on this page assume that you have completed the steps in Set up AI Agent Monitoring.

LLM Providers data integration can’t be saved

Saving the configuration for the LLM Providers data integration fails with the following error:
CODE 
Splunk Observability Cloud could not establish a connection with LLM provider. Review your authentication credentials and try again.
  1. Use the Splunk Observability Cloud main menu to navigate to Data Management > Deployed integrations. In the LLM Providers data integration:
    1. Verify that the Base URI url field isn't missing trailing / symbols. Review the following example URLs for common providers:
      LLM provider Example URLs Example models
      OpenAI https://api.openai.com/v1/

      • gpt-4o

      • gpt-4o-mini
      Google Gemini

      • https://generativelanguage.googleapis.com/v1/

      • OpenAI compatibility layer: https://generativelanguage.googleapis.com/v1beta/openai

        The OpenAI compatibility layer URL is in beta. While it simplifies integration, it may not support all Google-specific features (such as Grounding or specialized safety settings) available in the native v1 API.

      • gemini-2.5-pro

      • gemini-2.5-flash-lite
      Azure OpenAI

      https://<YOUR_AZURE_BASE_URL>/openai/v1/

      Replace <YOUR_AZURE_BASE_URL> with your Azure URL.

      • gpt-4o

      • gpt-4o-mini
      Anthropic

      https://api.anthropic.com/v1/

      • claude-haiku-4-5-20251001

      • claude-sonnet-4-6
    2. Verify that the API token you provided is active.
  2. Resolve the following potential issues with your model or instrumentation:
    Issue Description Solution
    Model is too large The model is too large, which causes evaluations to be generated too slowly and may result in timeout errors. Evaluations may incur higher costs in this scenario. Change to a smaller model.
    Model is too small A small model can yield inaccurate evaluations. Change to a larger model.
    Incompatible model An incompatible model can yield an incompatible output, which causes errors. Change to a compatible model.
    Instrumentation or telemetry issues Misconfigurations can prevent requests from being accepted or cause incompatible results. Troubleshoot your instrumentation to ensure that you're using the correct headers and connection.

Trace data is no longer being ingested

The LLM Providers data integration was correctly configured. You confirmed that trace data was being ingested because the APM > AI trace data page displayed traces with a value in the Quality column. However, the AI trace data page is no longer showing new trace data.

To resolve this issue, use the Splunk Observability Cloud main menu to navigate to Data Management > Deployed integrations. Proceed with the following steps to troubleshoot the LLM Providers data integration.

  1. Verify that the rate limit isn't exceeded and you haven't run out of tokens for your LLM provider.
  2. Verify that the LLM model provided in the Model name field is still supported by the provider.
  3. Verify that the API token you provided is active. Rotate the token if needed.
  4. Verify that the Number of evaluations you set aligns with your environment. Splunk Observability Cloud stops running evaluations once the number of evaluations per minute is reached.
  5. Verify that the Sampling rate isn't set to 0.