Switch your AI agent conversation data source

Switch between storing data in Splunk Observability Cloud or Splunk Enterprise/Splunk Cloud Platform.

This topic describes how to switch your AI agent conversation data source if you have already set up AI Agent Monitoring and configured a data source.

Feature support matrix for AI Agent Monitoring

You can store your AI conversation data in Splunk Observability Cloud or Splunk Enterprise/Splunk Cloud Platform.

Feature availability for AI Agent Monitoring depends on where you store your AI conversation data. For full access to AI Agent Monitoring features, store your AI conversation data in Splunk Observability Cloud.

The following table describes the features supported in each data source option.
Feature AI conversation data in Splunk Observability Cloud AI conversation data in Splunk Enterprise/Splunk Cloud Platform
Instrumentation-side evaluations No Yes
Platform-side evaluations Yes No
AI trace data page Yes No
AI overview page Yes No
Cost metrics Yes No
Risk metrics (with the Cisco AI Defense integration) Yes No

Switch from storing AI conversation data in Splunk Enterprise/Splunk Cloud Platform to Splunk Observability Cloud

This procedure is intended for users who have set up AI Agent Monitoring, stored AI conversation data in Splunk Enterprise or Splunk Cloud Platform logs, and enabled instrumentation-side evaluations.

Complete the following steps to switch to storing your AI conversation data in Splunk Observability Cloud.

You must meet the following requirements to switch your AI conversation data source to Splunk Observability Cloud:

  1. Switch the data source in the user interface:
    1. In the Splunk Observability Cloud main menu, select Settings > AI agent monitoring.
    2. Select Data source – Splunk Observability Cloud.
    3. Complete the on-screen steps to switch your data source. When prompted to refer to the documentation, proceed to the following steps.
      Note:

      Data source changes may take a few minutes to be reflected in the user interface.

  2. Turn off instrumentation-side evaluations:
    1. In your .env file, set OTEL_INSTRUMENTATION_GENAI_EVALS_EVALUATORS to none, off, or false. You can use any of these values to turn off instrumentation-side evaluations.

      If you don't complete this step, evaluations may be generated twice.

    2. In your .env file, set OTEL_INSTRUMENTATION_GENAI_EMITTERS to span_metric.
    3. Uninstall the instrumentation-side evaluation packages:
      CODE 
      pip uninstall splunk-otel-genai-evals-deepeval
pip uninstall splunk-otel-genai-emitters-splunk
    4. Restart your application.
  3. Configure the Splunk Distribution of the OpenTelemetry Collector to collect the correct data type:
    1. OpenTelemetry GenAI utility 0.1.14 and higher: In your .env file, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to SPAN_ONLY.
      Note:

      Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to SPAN_ONLY for platform-side evaluations and SPAN_AND_EVENT for instrumentation-side evaluations. If you don't complete this step, evaluations may be generated twice.

    2. OpenTelemetry GenAI utility 0.1.13 and lower: In your .env file, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE to SPAN_ONLY.
    3. If you don't plan to enable platform-side evaluations, restart your application.
  4. (Optional) Enable platform-side evaluations for AI agent responses.

After you set up AI Agent Monitoring and store your AI conversation data in Splunk Observability Cloud, you can:

Switch from storing AI conversation data in Splunk Observability Cloud to Splunk Enterprise/Splunk Cloud Platform

Note:

This alternative method requires a license for Splunk Enterprise or Splunk Cloud Platform, may incur additional resource costs, and limits feature availability. See the feature support matrix for details. For full access to Observability for AI features, store your AI conversation data in Splunk Observability Cloud.

This procedure is intended for users who have set up AI Agent Monitoring, stored AI conversation data in Splunk Observability Cloud, and enabled platform-side evaluations.

Complete the following steps to switch to storing your AI conversation data in Splunk Enterprise or Splunk Cloud Platform logs.

You must meet the following requirements to switch your AI conversation data source to Splunk Enterprise or Splunk Cloud Platform:

  • You have completed the steps on Set up AI Agent Monitoring.

  • You have a Splunk Enterprise or Splunk Cloud Platform license.

  • You have the admin role in Splunk Enterprise or Splunk Cloud Platform.

  • You have the admin role in Splunk Observability Cloud.

  1. Switch the data source in the user interface:
    1. In the Splunk Observability Cloud main menu, select Settings > AI agent monitoring.
    2. Select Data source – Splunk logs.
    3. Complete the on-screen steps to switch your data source. When prompted to refer to the documentation, proceed to the following steps.
      Note:

      Data source changes may take a few minutes to be reflected in the user interface.

  2. In your .env file, set OTEL_INSTRUMENTATION_GENAI_EMITTERS to span_metric_event.
  3. Configure the Splunk Distribution of the OpenTelemetry Collector to collect the correct data type:
    1. OpenTelemetry GenAI utility 0.1.14 and higher: In your .env file, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to SPAN_AND_EVENT.
      Note:

      Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to SPAN_ONLY for platform-side evaluations and SPAN_AND_EVENT for instrumentation-side evaluations. If you don't complete this step, evaluations may be generated twice.

    2. OpenTelemetry GenAI utility 0.1.13 and lower: In your .env file, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE to SPAN_AND_EVENT.
    3. If you don't plan to enable instrumentation-side evaluations, restart your application.
  4. To optionally enable instrumentation-side evaluations, see Store AI agent conversation data in Splunk Enterprise/Splunk Cloud Platform and enable instrumentation-side evaluations.

After you set up AI Agent Monitoring and store your AI conversation data in Splunk Enterprise or Splunk Cloud Platform logs, you can Monitor AI agents with Splunk APM.