Switch your AI agent conversation data source

Switch between storing data in Splunk Observability Cloud or Splunk Enterprise/Splunk Cloud Platform.

Attention:

Alpha features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this alpha feature available in its sole discretion and may discontinue it at any time. These documents are not yet publicly available and we ask that you keep such information confidential. Use of alpha features is subject to the Splunk Pre-Release Agreement for Hosted Services.

The following documentation links apply to users who want to store conversation data in Splunk Observability Cloud. Use the following links to navigate between the pages in this release:
This topic describes how to switch your AI agent conversation data source if you have already set up AI Agent Monitoring and configured a data source.

Feature support matrix for AI Agent Monitoring

You can store your AI conversation data in Splunk Observability Cloud or Splunk Enterprise/Splunk Cloud Platform. Feature availability for AI Agent Monitoring depends on where you store your AI conversation data. For full access to AI Agent Monitoring features, store your AI conversation data in Splunk Observability Cloud.

The following table describes the features supported in each data source option.
Feature AI conversation data in Splunk Observability Cloud AI conversation data in Splunk Enterprise/Splunk Cloud Platform
Instrumentation-side evaluations No Yes
Platform-side evaluations Yes No
AI trace data page Yes No
AI overview page Yes No
Cost metrics Yes No
Risk metrics (with the Cisco AI Defense integration) Yes No

Switch from storing AI conversation data in Splunk Enterprise/Splunk Cloud Platform to Splunk Observability Cloud

This procedure is intended for users who have set up AI Agent Monitoring, stored AI conversation data in Splunk Enterprise or Splunk Cloud Platform logs, and enabled instrumentation-side evaluations.

Complete the following steps to switch to storing your AI conversation data in Splunk Observability Cloud.

You must meet the following requirements to switch your AI conversation data source to Splunk Observability Cloud:

  • You have completed the steps on Set up AI Agent Monitoring (logs version).

  • You have the admin role in Splunk Observability Cloud.

  1. Switch the data source in the user interface:
    1. In the Splunk Observability Cloud main menu, select Settings > AI agent monitoring.
    2. Select Data source – Splunk Observability Cloud.
    3. Complete the on-screen steps to switch your data source. When prompted to refer to the documentation, proceed to the following steps.
      Note:

      Data source changes may take a few minutes to be reflected in the user interface.

  2. Turn off instrumentation-side evaluations:
    1. In your .env file, set OTEL_INSTRUMENTATION_GENAI_EVALS_EVALUATORS to none, off, or false. You can use any of these values to turn off instrumentation-side evaluations.

      If you don't complete this step, evaluations may be generated twice.

    2. In your .env file, set OTEL_INSTRUMENTATION_GENAI_EMITTERS to span_metric.
    3. Uninstall the instrumentation-side evaluation packages:
      CODE 
      pip uninstall splunk-otel-genai-evals-deepeval
pip uninstall splunk-otel-genai-emitters-splunk
    4. Restart your application.
  3. Configure the Splunk Distribution of the OpenTelemetry Collector to collect the correct data type:
    1. In your .env file, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE to SPAN.
      Note:

      Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE to SPAN for platform-side evaluations and SPAN_AND_EVENT for instrumentation-side evaluations. If you don't complete this step, evaluations may be generated twice.

    2. If you don't plan to enable platform-side evaluations, restart your application.
  4. (Optional) Enable platform-side evaluations for AI agent responses.

After you set up AI Agent Monitoring and store your AI conversation data in Splunk Observability Cloud, you can:

Switch from storing AI conversation data in Splunk Observability Cloud to Splunk Enterprise/Splunk Cloud Platform

Note:

This alternative method requires a license for Splunk Enterprise or Splunk Cloud Platform and may incur additional resource costs.

Storing AI conversation data in Splunk Enterprise or Splunk Cloud Platform limits feature availability. See the feature support matrix for details. For full access to Observability for AI features, store your AI conversation data in Splunk Observability Cloud.

This procedure is intended for users who have set up AI Agent Monitoring, stored AI conversation data in Splunk Observability Cloud, and enabled platform-side evaluations.

Complete the following steps to switch to storing your AI conversation data in Splunk Enterprise or Splunk Cloud Platform logs.

You must meet the following requirements to switch your AI conversation data source to Splunk Enterprise or Splunk Cloud Platform:

  • You have completed the steps on Set up AI Agent Monitoring (platform version).

  • You have a Splunk Enterprise or Splunk Cloud Platform license.

  • You have the admin role in Splunk Enterprise or Splunk Cloud Platform.

  • You have the admin role in Splunk Observability Cloud.

  1. Switch the data source in the user interface:
    1. In the Splunk Observability Cloud main menu, select Settings > AI agent monitoring.
    2. Select Data source – Splunk logs.
    3. Complete the on-screen steps to switch your data source. When prompted to refer to the documentation, proceed to the following steps.
      Note:

      Data source changes may take a few minutes to be reflected in the user interface.

  2. In your .env file, set OTEL_INSTRUMENTATION_GENAI_EMITTERS to span_metric_event.
  3. Configure the Splunk Distribution of the OpenTelemetry Collector to collect the correct data type:
    1. In your .env file, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE to SPAN_AND_EVENT.
      Note:

      Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT_MODE to SPAN for platform-side evaluations and SPAN_AND_EVENT for instrumentation-side evaluations. If you don't complete this step, evaluations may be generated twice.

    2. If you don't plan to enable instrumentation-side evaluations, restart your application.
  4. (Optional) Enable instrumentation-side evaluations for AI agent responses.

After you set up AI Agent Monitoring and store your AI conversation data in Splunk Enterprise or Splunk Cloud Platform logs, you can Monitor AI agents with Splunk APM.