Instrument your AI agent with OpenTelemetry Cisco AI Defense

Attention:

Alpha features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this alpha feature available in its sole discretion and may discontinue it at any time. These documents are not yet publicly available and we ask that you keep such information confidential. Use of alpha features is subject to the Splunk Pre-Release Agreement for Hosted Services.

The following documentation links apply to users who want to store conversation data in Splunk Observability Cloud. Use the following links to navigate between the pages in this release:

This topic explains how to instrument your AI agent with the OpenTelemetry Cisco AI Defense (opentelemetry-instrumentation-aidefense) library.

This instrumentation library can operate in either SDK mode or gateway mode:

Mode Description Use case
SDK mode Wraps cisco-aidefense-sdk methods Explicit security checks through inspect_prompt()
Gateway mode Extracts the X-Cisco-AI-Defense-Event-Id HTTP header in requests or responses and adds it (as the attribute gen_ai.security.event_id ) to the current LLM span LLM calls proxied through Cisco AI Defense Gateway

The library may attach these attributes to spans:

Attribute Type Description
gen_ai.security.event_id String Unique event ID from Cisco AI Defense. In gateway mode, the library sets only this attribute. Detailed risk information comes from server-side enrichment, not from the library.
gen_ai.request.model String cisco-ai-defense (SDK mode only)
gen_ai.system String aidefense (SDK mode only)
server.address String Cisco AI Defense API endpoint (SDK mode only)

The library may reference these environment variables:

Name Description
AI_DEFENSE_GATEWAY_URL Cisco AI Defense Gateway endpoint URL. Example: https://gateway.aidefense.security.cisco.com/{tenant}/connections/{conn}/v1
OTEL_INSTRUMENTATION_AIDEFENSE_GATEWAY_URLS Custom Cisco AI Defense Gateway URL patterns for auto-detection (comma-separated)
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT Set to true to capture full message content in spans
OTEL_EXPORTER_OTLP_ENDPOINT OTLP collector endpoint. Example: http://localhost:4317

  • Python v3.9 or newer

  • Supported LLMs and frameworks

  • opentelemetry-api v1.38.0 or newer

  • splunk-otel-util-genai v0.1.5 or newer

  • For SDK mode: cisco-aidefense-sdk v2.0.0 or newer

  • For gateway mode: httpx (for OpenAI, Cohere, Mistral) or boto3 (for AWS Bedrock)

  1. Install the instrumentation library:
    BASH 
    pip install splunk-otel-instrumentation-aidefense
  2. Install core OpenTelemetry dependencies:
    BASH 
    pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
  3. Set up the OpenTelemetry tracer provider.
    In your agent's entry point or initialization file, configure TracerProvider:
    PYTHON 
    from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Configure the OTLP exporter (pointing to your collector or Splunk endpoint)
exporter = OTLPSpanExporter(
    endpoint="https://your-splunk-endpoint:4317",
    headers={"x-sf-token": "your-splunk-access-token"}
)

# Set up the Tracer Provider
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
    Tip: Replace your-splunk-endpoint and your-splunk-access-token with your actual Splunk Observability Cloud credentials.
  4. Configure the library either in gateway mode or SDK mode.
  5. Add the AIDefenseInstrumentor().instrument() call to your AI agent:
    PYTHON 
    from opentelemetry.instrumentation.aidefense import AIDefenseInstrumentor

# Initialize and instrument
AIDefenseInstrumentor().instrument()
    Note: To ensure all interactions are captured add this call after other instrumentation like LangchainInstrumentor().instrument() but before your AI agent starts processing requests.
  6. Configure environment variables.
    Set the required environment variables for your instrumentation:
    BASH 
    export OTEL_SERVICE_NAME="my-ai-agent"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-splunk-endpoint:4317"
export OTEL_EXPORTER_OTLP_HEADERS="x-sf-token=your-splunk-access-token"
export OTEL_TRACES_EXPORTER="otlp"
  7. Run your AI agent.

    You can run your agent either directly or using the opentelemetry-instrument auto-instrumentation CLI:

    Direct run
    BASH 
    python your_agent.py
    Use the opentelemetry-instrument auto-instrumentation CLI
    BASH 
    opentelemetry-instrument python your_agent.py
  8. Verify telemetry data:
    1. On your Splunk Observability Cloud dashboard, navigate to APM > Traces.
    2. Search for your service name (for example, my-ai-agent).
    3. Confirm that AI agent interactions are appearing as spans or traces.

Putting it all together:

PYTHON 
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.aidefense import AIDefenseInstrumentor

# Step 1: Configure Tracer Provider
exporter = OTLPSpanExporter(
    endpoint="https://your-splunk-endpoint:4317",
    headers={"x-sf-token": "your-splunk-access-token"}
)
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Step 2: Instrument AI Agent
AIDefenseInstrumentor().instrument()

# Step 3: Your AI agent logic starts here
# ... your agent code ...

Troubleshoot your setup

Issue Resolution
No traces appearing Verify your endpoint URL and access token.
Import errors Confirm the package installed correctly via pip show opentelemetry-instrumentation-aidefense.
Missing spans Ensure instrument() is called before agent initialization.
Connection errors Check firewall rules and that port 4317 is open.