Configure zero-code instrumentation for Python AI applications

Instrument your backend Python AI applications to send metrics and traces to Splunk Observability Cloud.

Instrument your backend Python AI applications to send traces and metrics to Splunk Observability Cloud.

Zero-code instrumentation exports telemetry data using the OpenTelemetry GenAI utility and does not require changes to your application code. The instrumentation agent configures the source application to export data in a supported format to an OTLP endpoint, on either an OTLP receiver or the Splunk Observability Cloud backend.

For examples of applications using the instrumentation frameworks, see the examples directory of each instrumentation framework in splunk-otel-python-contrib/instrumentation-genai.

Prerequisite

Zero-code instrumentation requires Python 3.10 or higher.

Additional prerequisites may apply for certain instrumentation frameworks. For more details, see the respective section for your instrumentation framework.

Zero-code instrumentation integrations

CrewAI

Prerequisites

To instrument a CrewAI application, you must have access to an LLM provider, either through the OpenAI API key or OAuth2 credentials.

The CrewAI instrumentation captures workflow orchestration for crews, tasks, agents, and tools, but does not directly instrument LLM and embedding calls. For complete observability, including LLM call details such as token usage, model names, and latency, you must also install and enable provider-specific instrumentation packages:

  • OpenAI and Azure OpenAI: pip install opentelemetry-instrumentation-openai-v2

  • Anthropic: pip install opentelemetry-instrumentation-anthropic

  • Other providers: Check the splunk-otel-python-contrib/instrumentation-genai directory for available providers.

Without provider instrumentation, you'll see workflow structure but not the detailed LLM call spans shown in the trace view. These LLM call spans are represented by chat (OpenAI/LiteLLM) in the Expected Trace Structure diagram.

Steps

  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-crewai
  2. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

FastMCP

Prerequisites

To instrument a MCP server built with FastMCP, you must meet the fastmcp and util-genai version requirements specified in the Compatibility Matrix.

Steps

  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-fastmcp
    (Optional) To install both the instrumentation package and the latest version of FastMCP, run:
    CODE 
    pip install 'splunk-otel-instrumentation-fastmcp[instruments]'
  2. Run your server with the instrumentation:
    CODE 
    opentelemetry-instrument python your_mcp_server.py

  3. Configure the telemetry for your MCP transport mode. For more information on the configuration settings in this step, see Configure the Python agent for AI applications (0.1.14 and higher).

    stdio

    By default, stdio servers run in isolated environments and do not inherit the environment variables from your shell. You must explicitly pass required configurations to the server. When you spawn the FastMCP server from the Client, ensure that you pass OTEL_* in env and opentelemetry-instrument to command to StdioTransport as shown in the following command.

    1. Start the server (spawned from <your_fastmcp_client>.py):
      CODE 
      transport = StdioTransport(command="opentelemetry-instrument", args=["python", "your_fastmcp_server.py"], env=<ADD_OTEL_*_env_vars>)
client = Client(transport)

    2. Run the client:

      CODE 
      OTEL_SERVICE_NAME=<your_fastmcp_client_app> \
   opentelemetry-instrument python <your_fastmcp_client_app>.py # zero-code
      Note: In stdio mode, the server's OTEL_SERVICE_NAME must be set in the server process environment. The client and server get separate service names because they are separate processes.
    HTTP
    1. Set the following OpenTelemetry-specific environment variables in your server's .env file:
      CODE 
      OTEL_SERVICE_NAME=<your_fastmcp_server_app> \
OTEL_INSTRUMENTATION_GENAI_EMITTERS=span_metric
    2. Start the server:
      CODE 
      opentelemetry-instrument python <your_fastmcp_server_app>.py # zero-code
    3. Run the client:
      CODE 
      OTEL_SERVICE_NAME=<your_fastmcp_client_app> \
   opentelemetry-instrument python <your_fastmcp_client_app>.py # zero-code

GCP VertexAI

Prerequisites

To instrument a GCP VertexAI application, you must meet the following requirements.
  • You have access to a Google Cloud project with the Vertex AI API enabled and valid Google Cloud credentials.

  • Your application uses google-cloud-aiplatform version 1.64.0 or higher. Google has deprecated this module and plans to remove it on June 24, 2026.

Steps

  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-vertexai
  2. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

LangChain/LangGraph

Prerequisites

If your application is already instrumented with an OpenTelemetry SDK, you must upgrade your OpenTelemetry dependencies to version 1.38.0 or higher.

Note: Using the LangChain/LangGraph instrumentation along with a model provider-specific instrumentation (such as the OpenAI instrumentation) may lead to duplicated telemetry and evaluation results. To prevent this, model provider-specific instrumentation is suppressed by default with the SUPPRESS_LANGUAGE_MODEL_INSTRUMENTATION_KEY=true setting.

Steps

  1. Configure your LangChain/LangGraph application for AI Agent Monitoring:
    1. Set the agent_name for your Chains. This setting ensures that the instrumentation will promote your Chains to the AgentInvocation and evaluate your Chains with LLM-as-a-Judge evaluators. For example:
      JSON 
      agent = _create_react_agent(llm, tools=[]).with_config( 
  "metadata": { "agent_name": "coordinator"} 
)
    2. Set the workflow_name to promote the Chain or Graph to your workflow. For example:
      JSON 
      app = StateGraph(state).compile().with_config(metadata={"workflow_name": "multi_agent_travel_planner"})
  2. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-langchain
  3. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

LlamaIndex

Steps

  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-llamaindex
  2. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

OpenAI

Prerequisites

To instrument an OpenAI agent application, you must have an OpenAI API key (OPENAI_API_KEY) or access to an OpenAI-compatible LLM endpoint.

Steps
Note: By default, the LangChain instrumentation suppresses telemetry from the OpenAI instrumentation for calls made through LangChain. This setting is not configurable in the OpenAI instrumentation.
  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-openai
  2. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

OpenAI agents

Prerequisites

To instrument an OpenAI agent application, you must meet the following requirements.

  • You have an OpenAI API key (OPENAI_API_KEY) or access to an OpenAI-compatible LLM endpoint.

  • You have installed openai-agents-python SDK 0.3.3 or higher.

Steps

  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-openai-agents
  2. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

Weaviate

Steps

  1. Install the instrumentation package:
    CODE 
    pip install splunk-otel-instrumentation-weaviate
  2. Run your application with the instrumentation:
    CODE 
    opentelemetry-instrument python <your_app>.py

Next steps

To finish setting up AI Agent Monitoring, proceed to the next step in Set up AI Agent Monitoring.