Collect metrics and traces from a LiteLLM AI Gateway (LiteLLM Proxy)

Send metrics and traces from a LiteLLM AI Gateway (LiteLLM proxy) to the Splunk Distribution of the OpenTelemetry Collector.

You can configure the LiteLLM AI Gateway to send spans, metrics, and logs to the Splunk Distribution of the OpenTelemetry Collector.

The LiteLLM AI Gateway includes a built-in OpenTelemetry Integration module that enables an OpenTelemetry integration. The integration sends spans, metrics, and logs to the OTLP receiver.

To use this data integration, you must meet the following requirements:

  • You are using LiteLLM AI Gateway version 1.76.3 or higher. For information on LiteLLM AI Gateway versions, see the release notes in the LiteLLM GitHub repository.
    Note: LiteLLM AI Gateway is a dynamic project with complex dependencies. Upgrading your LiteLLM AI Gateway version may introduce breaking changes to your environment. Use tested versions before updating to the newest version.
  • You are using OpenTelemetry API and SDK version 1.25. Ensure that your Python packages match the following required LiteLLM AI Gateway packages:
    opentelemetry-api==1.25.0 
opentelemetry-sdk==1.25.0 
opentelemetry-exporter-otlp==1.25.0
  1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:
  2. Add the following environment variables to the configuration of your LiteLLM AI Gateway service:
    LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true  
LITELLM_OTEL_INTEGRATION_ENABLE_METRICS=true  
OTEL_SERVICE_NAME=<yourServiceName>  
OTEL_EXPORTER_OTLP_PROTOCOL=<grpc, http/protobuf, or http/json> 
OTEL_EXPORTER_OTLP_ENDPOINT=<yourCollectorEndpoint>:<yourCollectorPort>  
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA
  3. Add the following code in your LiteLLM AI Gateway config.yaml file:
    litellm_settings:  
  callbacks: ["otel"]
  4. Run the LiteLLM AI Gateway with the startup command:
    'litellm --config config.yaml'

Configuration settings

Learn about the configuration settings for the OTLP receiver.

To view the configuration options for the OTLP receiver, see Settings.

Metrics

The following metrics are available for LiteLLM AI Gateways. These metrics fall under the default metric category in Splunk Observability Cloud.

For more information on these metrics, see Semantic conventions for generative AI metrics in the OpenTelemetry documentation.

Metric nameData typeMetric typeUnitDescription
gen_ai.client.operation.durationfloathistogramsecondsThe duration of the GenAI operation.
gen_ai.client.token.usageinthistogramcountMeasures the number of input (prompt) and output (completions) tokens used.
gen_ai.client.token.costfloathistogramUSDMeasures the cost of the tokens used.

Attributes

The following resource attributes are available for LiteLLM AI Gateways.

Splunk Observability Cloud only displays attributes prefixed with metadata if the metadata is set in the LiteLLM administration portal or config.yaml file. For instructions, see the following pages in the LiteLLM documentation:

Attribute nameTypeDescription
gen_ai.operation.namestring

The name of the operation being performed.

Possible values:

  • chat

  • generate_content

  • text_completion

gen_ai.systemstringThe name of the GenAI system that produced the tokens.
gen_ai.request.modelstringThe name of the GenAI model a request is being made to.
gen_ai.frameworkstringThe name of the library used to interact with the GenAI system.
gen_ai.token.typestring

The type of token being counted. For example, input or output.

This attribute is only applicable for the gen_ai.client.token.usage metric.

metadata.user_api_key_hashstringThe user API key as a hash value. For identifying user keys in metrics.
metadata.user_api_key_aliasstringThe user API key alias as defined in the LiteLLM AI Gateway.
metadata.user_api_key_team_idstringThe ID of the team the user API key is associated with.
metadata.user_api_key_user_idstringThe ID of the user of the API key.
metadata.user_api_key_team_aliasstringThe alias of the team the user API key is associated with.
metadata.requester_ip_addressstringThe IP address of the requester, if available.
metadata.requester_metadatastringAny metadata provided by LiteLLM AI Gateway about the requester.
metadata.applied_guardrailsstringAny guardrails applied to this request via LiteLLM AI Gateway.