Monitor LangChain Applications

Splunk AppDynamics Python Agent helps you to monitor GenAI apps built using the LangChain framework. When you instrument the app with Python Agent, it captures the following:

  • Exit calls to vector databases: pgvector and chroma.
  • Prompts and responses in the exit call.
  • Counts of documents retrieved, similarity search score, and errors in vector databases
  • Calls per minute

To monitor LLMs using LangChain, such as Ollama, the Python Agent records these metrics:

  • Tokens
  • Prompt count
  • Embedding queries count

Prerequisites

Ensure to set the enable-langchain flag to true in /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].

Python agent supports the following versions of LangChain LLMs in Splunk AppDynamics.

Component Version
langchain <= 0.2.11
langchain-ollama <= 0.2.0
langchain-chroma <= 0.1.1
langchain-postgres <= 0.0.12
chromadb<= 0.5.20
pgvector<= 0.2.5

Monitor LangChain Ollama APIs

To monitor Ollama calls, the Python Agent reports these metrics:

  • Input Tokens
  • Output Tokens
  • Prompt count
  • Embedding queries count
  • Errors
Note:

For Token Metrics, ensure to install transformers Python library. See transformers.

pip install transformers

Create a Custom Dashboard to Monitor Ollama APIs

  1. Download Custom_dashboard_script.zip It contains:
    DirectoryDescription
    templatesContains json templates to create the dashboard files.
    dashboardDashboard json files are created here. It contains the sample json files.
    main.pyPython script to create the dashboards.
    readme.mdDescription of the solution.
  2. Run the following command:
    cd Custom_dashboard_script
python3 main.py

    The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.

  3. Based on your selected dashboard type, you will be prompted to specify the following dashboard detail.
  4. Import the dashboard in the Controller UI. See Create Custom Dashboard
    Tip: Navigate to Tiers & Nodes > My Dashboards and create a dashboard for the selected application by copying and pasting widgets from the existing dashboard for easier navigation.
Example DashboardCustom Dashboard to Monitor Ollama APIs

Monitor Vectorstores

When you instrument the app with Python Agent, it captures the following for Vectorstores:

  • Exit calls to vector databases: pgvector and chroma.
  • Counts of documents retrieved, similarity search score, and errors in vector databases
  • Calls per minute

To capture Vectorstores request and response in snapshot exit call detail, set the enable-genai-data-capture flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].