Collect metrics and traces from a LiteLLM proxy

Send metrics and traces from a LiteLLM proxy service to the Splunk Distribution of the OpenTelemetry Collector.

You can configure the LiteLLM proxy service to collect metrics and traces and send them to the Splunk Distribution of the OpenTelemetry Collector. The LiteLLM instrumentation sends metrics and traces to the OTLP receiver.

To use the LiteLLM proxy instrumentation to push metrics and traces to the OTLP receiver, you must install and update the OpenTelemetry Python distribution package: 
pip3 install 'opentelemetry-distro[otlp]'
  1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:
  2. Add the following environment variables to the configuration of your LiteLLM proxy service:
    LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true LITELLM_OTEL_INTEGRATION_ENABLE_METRICS=true 
OTEL_SERVICE_NAME=<yourServiceName>  
OTEL_EXPORTER_OTLP_PROTOCOL=grpc  
OTEL_EXPORTER_OTLP_ENDPOINT=<yourCollectorEndpoint>:<yourCollectorPort>
  3. Add the following code in your LiteLLM proxy configuration YAML file:
    litellm_settings:  
  callbacks: ["otel"]
  4. Run the LiteLLM proxy with the OpenTelemetry instrumentation by adding the prefix 'opentelemetry-instrument' to the LiteLLM proxy startup command:
    'litellm --config config.yaml'

Configuration settings

Learn about the configuration settings for the OTLP receiver.

To view the configuration options for the OTLP receiver, see Settings.

Metrics

The following metrics are available for LiteLLM proxy services. These metrics fall under the default metric category. For more information on these metrics, see Semantic conventions for generative AI metrics in the OpenTelemetry documentation.

Metric nameData typeMetric typeUnitDescription
gen_ai.client.operation.durationfloathistogramsecondsThe duration of the GenAI operation.
gen_ai.client.token.usageinthistogramcountMeasures the number of input (prompt) and output (completions) tokens used.

Attributes

The following resource attributes are available for LiteLLM proxy services.

Attribute nameTypeDescription
gen_ai.operation.namestring

The name of the operation being performed.

Possible values:

  • chat

  • generate_content

  • text_completion

gen_ai.systemstringThe name of the GenAI system that produced the tokens.
gen_ai.request.modelstringThe name of the GenAI model a request is being made to.
gen_ai.frameworkstringThe name of the library used to interact with the GenAI system.
gen_ai.token.typestring

The type of token being counted. For example, input or output.

This attribute is only applicable for the gen_ai.client.token.usage metric.