Store AI agent conversation data in Splunk Enterprise/Splunk Cloud Platform and enable instrumentation-side evaluations

This method requires a license for Splunk Enterprise/Splunk Cloud Platform, may incur additional resource costs, and limits feature availability.

Note:

This alternate method is provided for users who want to store their AI agent conversation data in Splunk Enterprise or Splunk Cloud Platform and enable instrumentation-side evaluations for AI agent responses.

This data storage method requires a license for Splunk Enterprise or Splunk Cloud Platform, may incur additional resource costs, and limits feature availability. See the Feature support matrix for more details.

For increased data security, you can store your AI agent conversation data in Splunk Enterprise or Splunk Cloud Platform. This data storage option supports instrumentation-side evaluations for AI agent responses.

Instrumentation-side evaluations are built into the instrumentation frameworks for AI applications. Splunk instrumentation frameworks trigger evaluations performed by your LLM based on DeepEval, an open-source evaluation framework. The Splunk Distribution of the OpenTelemetry Collector sends the evaluation results to Splunk Observability Cloud. Splunk Observability Cloud receives and displays evaluation results as quality scores, but does not have visibility into your interaction inputs or outputs.

Instrumentation-side evaluations are only supported if you're storing conversation data in Splunk Enterprise or Splunk Cloud Platform. You can only enable instrumentation-side evaluations or platform-side evaluations, not both at the same time.

Feature support matrix for AI Agent Monitoring

You can store your AI conversation data in Splunk Observability Cloud or Splunk Enterprise/Splunk Cloud Platform.

Feature availability for AI Agent Monitoring depends on where you store your AI conversation data. For full access to AI Agent Monitoring features, store your AI conversation data in Splunk Observability Cloud.

The following table describes the features supported in each data source option.
Feature AI conversation data in Splunk Observability Cloud AI conversation data in Splunk Enterprise/Splunk Cloud Platform
Instrumentation-side evaluations No Yes
Platform-side evaluations Yes No
AI trace data page Yes No
AI overview page Yes No
Cost metrics Yes No
Risk metrics (with the Cisco AI Defense integration) Yes No

To enable instrumentation-side evaluations, you must meet the following requirements:

  • You have a Splunk Enterprise or Splunk Cloud Platform license.

  • You have the admin role in Splunk Enterprise or Splunk Cloud Platform.

  • You have the admin role in Splunk Observability Cloud.

  • You have completed the steps on Collect traces and metrics from AI agents and applications.

  • You aren't storing AI agent conversation data in Splunk Observability Cloud, and you haven't enabled platform-side evaluations.

  1. (Optional) Collect logs and events with the Collector for Kubernetes.
  2. Set the following environment variables in your .env file. For more information on these environment variables and their supported values, see Configure the Python agent for AI applications (0.1.14 and higher).
    CODE 
    # Emitters (span_metric_event for full telemetry, splunk for Splunk-specific features)
OTEL_INSTRUMENTATION_GENAI_EMITTERS=span_metric_event,splunk

# Content Capture
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_AND_EVENT

# Logs
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

# Metrics
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta
    Note:

    Enabling GenAI content capture with one of the following settings may cause performance issues:

    • OpenTelemetry GenAI utility version 0.1.14 and higher: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT as SPAN_AND_EVENT, SPAN_ONLY, or EVENT_ONLY

    • OpenTelemetry GenAI utility version 0.1.13 and lower: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

    This issue may arise when content captured in input and output gen_ai span attributes is larger than Splunk Observability Cloud and AI Agent Monitoring backend limits. For more information, see Recording content on attributes in the OpenTelemetry documentation.

  3. Collect logs and events from AI agents and applications:
    Splunk Enterprise

    1. Create an events index in Splunk Enterprise to store and process your data. For instructions, see Create events indexes.

    2. Create an HTTP Event Collector (HEC) token in Splunk Enterprise. The HEC token enables you to send data and application events to your Splunk index over HTTP protocol using token-based authentication.

      For requirements and instructions, see Configure HTTP Event Collector on Splunk Enterprise.

    3. Set up Log Observer Connect. For instructions, see Set up Log Observer Connect for Splunk Enterprise.

    4. Configure the Splunk HEC exporter to allow the OpenTelemetry Collector to send logs to Splunk HEC endpoints.

      The following example shows a Splunk HEC exporter instance configured for a logs pipeline in the Collector configuration file:

      YAML 
      exporters:
  # ...
  splunk_hec:
    token: "<hec-token>"
    endpoint: "<hec-endpoint>"
    # Source. See https://docs.splunk.com/Splexicon:Source
    source: "otel"
    # Source type. See https://docs.splunk.com/Splexicon:Sourcetype
    sourcetype: "otel"

# ...

      Next, add the exporter to the services section of your configuration file:

      YAML 
      service:
  # ...
  pipelines:
    logs:
      receivers: [fluentforward, otlp]
      processors:
      - memory_limiter
      - batch
      - resourcedetection
      exporters: [splunk_hec]
    Splunk Cloud Platform

    1. Create an events index in Splunk Cloud Platform to store and process your data. For instructions, see Create a Splunk Cloud Platform events index.

    2. Create an HTTP Event Collector (HEC) token in Splunk Cloud Platform. The HEC token enables you to send data and application events to your Splunk index over HTTP protocol using token-based authentication.

      For requirements and instructions, see Manage HTTP Event Collector (HEC) tokens in Splunk Cloud Platform.

    3. Set up Log Observer Connect. For instructions, see Set up Log Observer Connect for Splunk Cloud Platform.

    4. Configure the Splunk HEC exporter to allow the OpenTelemetry Collector to send logs to Splunk HEC endpoints.

      The following example shows a Splunk HEC exporter instance configured for a logs pipeline in the Collector configuration file:

      YAML 
      exporters:
  # ...
  splunk_hec:
    token: "<hec-token>"
    endpoint: "<hec-endpoint>"
    # Source. See https://docs.splunk.com/Splexicon:Source
    source: "otel"
    # Source type. See https://docs.splunk.com/Splexicon:Sourcetype
    sourcetype: "otel"

# ...

      Next, add the exporter to the services section of your configuration file:

      YAML 
      service:
  # ...
  pipelines:
    logs:
      receivers: [fluentforward, otlp]
      processors:
      - memory_limiter
      - batch
      - resourcedetection
      exporters: [splunk_hec]
  4. To correlate AI user conversations with your APM traces, configure Splunk Observability Cloud to query your Splunk Enterprise or Splunk Cloud Platform connection and index:
    1. In Splunk Observability Cloud, use the main menu to select Settings > AI agent monitoring.
    2. Select Data source – Splunk logs.
    3. Under Connection selection, select your Splunk Enterprise or Splunk Cloud Platform instance.
    4. For Index selection, select the events index that you created.
    5. Select Apply.
  5. Enable instrumentation-side evaluations by installing the required packages and setting the environment variables. For more information on the configuration settings in this step, see Configure the Python agent for AI applications (0.1.14 and higher).
    1. Install the packages:
      CODE 
      pip install splunk-otel-genai-evals-deepeval
pip install splunk-otel-genai-emitters-splunk
    2. To send evaluation results to Splunk Observability Cloud, set the following environment variables in your .env file.
      CODE 
      # Evaluations
OTEL_INSTRUMENTATION_GENAI_EVALS_RESULTS_AGGREGATION=true
OTEL_INSTRUMENTATION_GENAI_EMITTERS_EVALUATION=replace-category:SplunkEvaluationResults
    3. By default, the instrumentation frameworks run evaluations in the same process as your application. LLM calls from evaluation frameworks such as DeepEval are instrumented alongside application telemetry.

      To run evaluations in a child process with the OpenTelemetry SDK deactivated and prevent evaluation LLM calls from polluting application telemetry, set OTEL_INSTRUMENTATION_GENAI_EVALS_SEPARATE_PROCESS=true in your .env file.
      Note: This setting is required when evaluations are enabled for the OpenAI instrumentation. This setting is optional for all other instrumentation frameworks that have evaluations enabled.
    4. To enable LLM-as-a-Judge evaluations with DeepEval, use one of the following options.

      1. To use OpenAI as the LLM provider for evaluations, set the OPENAI_API_KEY in your .env file. This is the default option.

      2. To use a custom LLM provider, run pip install litellm to install the LiteLLM dependencies and set the following environment variables in your .env file. Route evaluations through your own LLM provider instead of OpenAI.

        CODE 
        DEEPEVAL_LLM_BASE_URL=https://<your-llm-gateway>/openai/v1
DEEPEVAL_LLM_MODEL=gpt-4o-mini
DEEPEVAL_LLM_PROVIDER=openai
DEEPEVAL_LLM_CLIENT_APP_NAME=<your-app-key>
DEEPEVAL_FILE_SYSTEM=READ_ONLY

# For providers that don't require OAuth2
DEEPEVAL_LLM_API_KEY=<your-api-key>

# For providers that require OAuth2
DEEPEVAL_LLM_TOKEN_URL=https://<your-identity-provider>/oauth2/token
DEEPEVAL_LLM_CLIENT_ID=<your-oauth2-client-id>
DEEPEVAL_LLM_CLIENT_SECRET=<your-oauth2-client-secret>

# Add custom headers as JSON to LLM API requests
DEEPEVAL_LLM_EXTRA_HEADERS='{"system-code": "APP-123", "x-custom-header": "value
    5. (Optional) To prevent DeepEval telemetry from appearing in the trace waterfall view when monitoring AI agents, set DEEPEVAL_TELEMETRY_OPT_OUT="YES" in your .env file.
    6. Restart your application.
  6. Verify that your data is being ingested by using the Splunk Observability Cloud main menu to navigate to APM > AI agents. If you don't see data on this page:
    1. Ensure that your Log Observer Connect index is set to the index that contains your AI trace data.
    2. Troubleshoot AI Agent Monitoring.

After you set up AI Agent Monitoring, you can Monitor AI agents with Splunk APM.