Set up AI Agent Monitoring

Get data in, enable platform-side evaluations for AI agent responses, and start monitoring your AI agents and applications.

Complete the following high-level steps to set up AI Agent Monitoring.

  1. Collect traces and metrics from AI agents and applications.

  2. (Optional) Enable platform-side evaluations for AI agent responses.

  3. (Optional) Set up the Cisco AI Defense integration.

  4. Verify that your data is being ingested.

Collect traces and metrics from AI agents and applications

Complete the following steps to collect traces and metrics from your AI agents and applications.

  1. Deploy the Splunk Distribution of OpenTelemetry Collector on the hosts that your applications are running on. You can either use the guided setup wizards in Splunk Observability Cloud or install the Collector programmatically.

    Guided setup wizards

    Complete the following steps to use the guided setup wizards to deploy the Collector on a host.

    1. In the Splunk Observability Cloud main menu, select Data Management > Available integrations.

    2. Select Deploy OpenTelemetry collectors > Deploy Splunk OpenTelemetry Collector for other environments.

    3. Follow the on-screen instructions to deploy the Collector on your host.

    Advanced installation

    To programmatically install the Collector, see:

  2. Histogram metrics are required to display data on AI Agent Monitoring pages.

    To send histogram data to Splunk Observability Cloud with the SignalFx exporter, set send_otlp_histograms: true in your Collector values.yaml file. For example:

    YAML 
    exporters:
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    api_url: "${SPLUNK_API_URL}"
    ingest_url: "${SPLUNK_INGEST_URL}"
    sync_host_metadata: true
    correlation:
    send_otlp_histograms: true

  3. If the Python agent is installed in your Kubernetes cluster, configure the Kubernetes Downward API to expose environment variables to Kubernetes resources.

    The following example shows how to update a deployment to expose environment variables by adding the agent configuration under the .spec.template.spec.containers.env section:
    YAML 
    apiVersion: apps/v1
kind: Deployment
spec:
  selector:
    matchLabels:
      app: your-application
  template:
    spec:
      containers:
        - name: myapp
          env:
            - name: SPLUNK_OTEL_AGENT
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://$(SPLUNK_OTEL_AGENT):4317"
            - name: OTEL_SERVICE_NAME
              value: "<serviceName>"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "deployment.environment=<environmentName>"

    To optionally configure the Python agent to send telemetry to Splunk Observability Cloud using other methods, see Instrument your Python application for Splunk Observability Cloud.

  4. Set the following environment variables in your .env file. For more information on these environment variables and their supported values, see Configure the Python agent for AI applications (0.1.14 and higher).
    CODE 
    # Emitters (span_metric for full telemetry)
OTEL_INSTRUMENTATION_GENAI_EMITTERS=span_metric

# Content Capture
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY

# Metrics
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
    Note:

    Enabling GenAI content capture with one of the following settings may cause performance issues:

    • OpenTelemetry GenAI utility version 0.1.14 and higher: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT as SPAN_AND_EVENT, SPAN_ONLY, or EVENT_ONLY

    • OpenTelemetry GenAI utility version 0.1.13 and lower: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

    This issue may arise when content captured in input and output gen_ai span attributes is larger than Splunk Observability Cloud and AI Agent Monitoring backend limits. For more information, see Recording content on attributes in the OpenTelemetry documentation.

  5. Instrument or translate data from AI applications using one or more of the following options:

    Option Description
    Zero-code instrumentation Exports telemetry data without changes to your application's source code.
    Code-based instrumentation Exports telemetry data and requires modifying your application's source code.
    Translate data from third-party instrumentation libraries Converts data from applications already instrumented with supported third-party libraries and sends the data to Splunk Observability Cloud.
    To troubleshoot the Splunk Distribution of the OpenTelemetry Collector, see Troubleshoot the Collector.

(Optional) Enable platform-side evaluations for AI agent responses

Evaluation is the process of testing a large-language model (LLM) to assess the bias, hallucination, relevance, sentiment, or toxicity of its outputs. Splunk Observability Cloud displays evaluation results so you can monitor the quality of your AI agent responses.

Platform-side evaluations are performed within Splunk Observability Cloud after conversation data has been ingested. To enable platform-side evaluations, you must store your AI agent conversation data in Splunk Observability Cloud.
Note:

If you want to store your AI conversation data in Splunk Enterprise or Splunk Cloud Platform and monitor evaluation results, you can enable instrumentation-side evaluations.

This alternate data storage method requires a license for Splunk Enterprise or Splunk Cloud Platform, may incur additional resource costs, and limits feature availability. For instructions, see Store AI agent conversation data in Splunk Enterprise/Splunk Cloud Platform and enable instrumentation-side evaluations.

Complete the following steps to enable platform-side evaluations.
  1. Set up the LLM data integration:

    1. In the Splunk Observability Cloud main menu, select Data Management > Available integrations.

    2. Search for and select LLM Providers.

    3. Follow the on-screen instructions to set up the data integration. After you set up the data integration, evaluations may take a few minutes to begin running.

      If you receive an error when you try to save your configuration, see the troubleshooting topic LLM Providers data integration can't be saved.

  2. Verify that the LLM Providers data integration was correctly configured and trace data is being ingested:

    1. In the Splunk Observability Cloud main menu, select APM > AI trace data.

    2. Add a filter to display data beginning from after you configured the LLM Providers data integration.

    3. If you don't see data on this page, see the troubleshooting topic Troubleshoot data ingestion for AI Agent Monitoring.

(Optional) Set up the Cisco AI Defense integration

Set up the Cisco AI Defense integration to enable security risk metrics for AI agents.

For more information about the integration, see Introduction to Splunk AI Security Monitoring. For setup instructions, see Set up an integration with Cisco AI Defense.

Verify that your data is being ingested

Verify that your data is being ingested by using the Splunk Observability Cloud main menu to navigate to APM > AI agents.

If you don't see data on this page:

Next steps

After you set up AI Agent Monitoring, you can: