Monitor LLM services with Splunk APM

Learn how to monitor LLM services with Splunk APM.

Attention: Beta features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this beta feature available in its sole discretion and may discontinue it at any time. Use of beta features is subject to the Splunk General Terms.
If you have set up Splunk Application Performance Monitoring (APM), you can instrument and monitor the performance of large-language model (LLM) services with the service map and trace view.

For more information about what you can do with Splunk APM, see What can you do with Splunk APM.

View LLM services on the service map

Learn how to view LLM services on the APM service map.

On the APM service map, you can filter services to view or hide interactions with and within LLM services. To filter only by LLM services, access the service map and select AI services in the list of service types in the filter menu. Your LLM services are categorized as inferred services in Splunk APM. For more information, see Filter the service map.

Analyze LLM service traces in the trace view

Learn how to analyze LLM service traces in the trace view.

To access the trace view, navigate to the Trace Analyzer and select a Trace ID. Select the AI Interactions tab to view spans for your LLM services and their attributes. The spans are filtered using the gen_ai.operation.name attribute. To view definitions for the other span attributes, see Spans in the OpenTelemetry documentation.

The following screenshot displays an example of the content in the AI Interactions tab in the trace view.A screenshot of the AI Interactions tab in the APM trace view.

Additionally, view a parsed log summary of your LLM service responses in order to observe the quality of your LLM service responses on the AI Events tab. An outlined chip appears next to the LLM services to differentiate these services with others in your environment.
Note: Collection of prompts and responses is disabled by default, but can be enabled using the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT environment variable in the OpenTelemetry Collector and the Splunk Distribution of the OpenTelemetry Collector. Prompts and responses may contain sensitive or personally identifiable information (PII). When collection is enabled, PII data is sent to Splunk Cloud Platform as log events, allowing you to store sensitive data within your security boundaries. You can also use Splunk Cloud Platform’s PII masking mechanisms for enhanced data protection. You must set up a connection with Splunk Log Observer Connect in order to view logs events in conjunction with traces. To set up a connection, see Set up Log Observer Connect.

Link data links to AI navigators in Infrastructure Monitoring

Learn how to link data links to AI navigators in Infrastructure Monitoring.

You can use data links to link APM views with the LangChain and OpenAI navigators. To configure a data link, see Create a global data link and link to a Splunk Observability Cloud Navigator.