Set up AI Agent Monitoring

Get data in, enable platform-side evaluations for AI agent responses, and start monitoring your AI agents and applications.

Attention:

Alpha features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this alpha feature available in its sole discretion and may discontinue it at any time. These documents are not yet publicly available and we ask that you keep such information confidential. Use of alpha features is subject to the Splunk Pre-Release Agreement for Hosted Services.

The following documentation links apply to users who want to store conversation data in Splunk Observability Cloud. Use the following links to navigate between the pages in this release:

Complete the following high-level steps to set up AI Agent Monitoring.

  1. Collect traces and metrics from AI agents and applications.
  2. (Optional) Enable platform-side evaluations for AI agent responses.
  3. (Optional) Set up the Cisco AI Defense integration.
  4. Verify that your data is being ingested.

Collect traces and metrics from AI agents and applications

Complete the following steps to collect traces and metrics from your AI agents and applications.

  1. Deploy the Splunk Distribution of OpenTelemetry Collector on the hosts that your applications are running on. You can either use the guided setup wizards in Splunk Observability Cloud or install the Collector programmatically.
    Guided setup wizards

    Complete the following steps to use the guided setup wizards to deploy the Collector on a host.

    1. In the Splunk Observability Cloud main menu, select Data Management > Available integrations.

    2. Select Deploy OpenTelemetry collectors > Deploy Splunk OpenTelemetry Collector for other environments.

    3. Follow the on-screen instructions to deploy the Collector on your host.

    Advanced installation

    To programmatically install the Collector, see:

  2. If the Python agent is installed in your Kubernetes cluster, configure the Kubernetes Downward API to expose environment variables to Kubernetes resources.
    The following example shows how to update a deployment to expose environment variables by adding the agent configuration under the .spec.template.spec.containers.env section:
    YAML 
    apiVersion: apps/v1
kind: Deployment
spec:
  selector:
    matchLabels:
      app: your-application
  template:
    spec:
      containers:
        - name: myapp
          env:
            - name: SPLUNK_OTEL_AGENT
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://$(SPLUNK_OTEL_AGENT):4317"
            - name: OTEL_SERVICE_NAME
              value: "<serviceName>"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=<environmentName>"

    To optionally configure the Python agent to send telemetry to Splunk Observability Cloud using other methods, see Instrument your Python application for Splunk Observability Cloud.

  3. Instrument or translate data from AI applications using one or more of the following options:
    Option Description
    Zero-code instrumentation Exports telemetry data without changes to your application's source code.
    Code-based instrumentation Exports telemetry data and requires modifying your application's source code.
    Translate data from third-party instrumentation libraries Converts data from applications already instrumented with supported third-party libraries and sends the data to Splunk Observability Cloud.
    To troubleshoot the Splunk Distribution of the OpenTelemetry Collector, see Troubleshoot the Collector.
  4. Histogram metrics are required to display data on the AI overview, AI agents, and AI trace data pages.

    To send histogram data to Splunk Observability Cloud with the SignalFx exporter, set send_otlp_histograms: true in your Collector values.yaml file. For example:

    YAML 
    exporters:
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    api_url: "${SPLUNK_API_URL}"
    ingest_url: "${SPLUNK_INGEST_URL}"
    sync_host_metadata: true
    correlation:
    send_otlp_histograms: true

(Optional) Enable platform-side evaluations for AI agent responses

Evaluation is the process of testing a large-language model (LLM) to assess the bias, hallucination, relevance, sentiment, or toxicity of its outputs. Splunk Observability Cloud displays evaluation results so you can monitor the quality of your AI agent responses.

Platform-side evaluations are performed within Splunk Observability Cloud after AI agent conversation data has been ingested. To enable platform-side evaluations, you must store your AI agent conversation data in Splunk Observability Cloud. Complete the following steps to enable platform-side evaluations.

  1. If you instrumented your application with Splunk zero-code or code-based instrumentation, set the following environment variables in your .env file. For more information on the configuration settings in this step, see Configure the Python agent for AI applications.
    CODE 
    # Emitters (span_metric for full telemetry)
OTEL_INSTRUMENTATION_GENAI_EMITTERS=span_metric

# Content Capture
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE=SPAN

# Metrics
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
  2. Restart your application.
  3. Set up the LLM data integration:
    1. In the Splunk Observability Cloud main menu, select Data Management > Available integrations.
    2. Search for and select LLM Providers.
    3. Follow the on-screen instructions.

      If you receive an error when you try to save your configuration, see the troubleshooting topic LLM Providers data integration can't be saved.

  4. Verify that the LLM Providers data integration was correctly configured and trace data is being ingested:
    1. In the Splunk Observability Cloud main menu, select APM > AI trace data.
    2. Add a filter to display data beginning from after you configured the LLM Providers data integration.
    3. If your LLM Providers data integration wasn't configured correctly, see the troubleshooting topics LLM Providers data integration can’t be saved and Trace data is no longer being ingested.

Verify that your data is being ingested

Verify that your data is being ingested by using the Splunk Observability Cloud main menu to navigate to APM > AI agents.

If you don't see data on this page: