Monitor LangChain Applications

Splunk AppDynamics Python Agent helps you to monitor GenAI apps built using the LangChain framework. When you instrument the app with Python Agent, it captures the following:

  • Exit calls to vector databases: pgvector and chroma.
  • Prompts and responses in the exit call.
  • Counts of documents retrieved, similarity search score, and errors in vector databases
  • Calls per minute

To monitor LLMs using LangChain, such as Ollama, the Python Agent records these metrics:

  • Input Tokens
  • Output Tokens
  • Time to first token metric (ms)
  • Time per output token (ms)
  • Average Response Time (ms)
  • Prompt count
  • Embedding queries count
  • Errors

Prerequisites

Ensure to set the enable-langchain flag to true in /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].

LangChain Ollama API のモニタリング

Ollama コールをモニターするために、Python エージェントは次のメトリックをレポートします。

  • 入力トークン = 0.5$
  • 出力トークン
  • 最初のトークンメトリックまでの時間（ミリ秒）
  • 出力トークンあたりの時間（ミリ秒）
  • 平均応答時間（ms）
  • プロンプト数
  • 組み込みクエリ数
  • エラー
注:

トークンメトリックを取得するには、transformers Python ライブラリをインストールしてください。「Transformers」を参照してください。

pip install transformers

メトリックパスの例：Agent|Langchain|LLM|llama3.2_latest|Average Response Time (ms))

Create a Custom Dashboard to Monitor Ollama APIs

  1. Download Custom_dashboard_script.zip It contains:
    DirectoryDescription
    templatesContains json templates to create the dashboard files.
    dashboardDashboard json files are created here. It contains the sample json files.
    main.pyPython script to create the dashboards.
    readme.mdDescription of the solution.
  2. Run the following command:
    cd Custom_dashboard_script
python3 main.py

    The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.

  3. Based on your selected dashboard type, you will be prompted to specify the following dashboard detail.
  4. Import the dashboard in the Controller UI. See Create and Manage Custom Dashboards and Templates
    ヒント: Navigate to Tiers & Nodes > My Dashboards and create a dashboard for the selected application by copying and pasting widgets from the existing dashboard for easier navigation.
Example DashboardCustom Dashboard to Monitor Ollama APIs

Monitor LangChain Vectorstores

When you instrument the app with Python Agent, it captures the following for Vectorstores:

  • For langchain_community and langchain_{vendor} vectorstores, Python Agent captures the following metrics:
    • Search Score
    • Average Response Time (ms)
    • Errors
    • Calls
    • Vector Insertion Count
    • Vector Deletion Count
    Note:

    By default, capture the following metrics for langchain_{vendor} vectorstores:

    • langchain_milvus
    • langchain_chroma
    • langchain_postgres

    However, you can set the langchain-vectorstores-instrumented-modules flag to capture the metrics from the specific LangChain vendor in /path/to/appdynamics.cfg. For more information, see [instrumentation].

  • Exit calls to vector databases: pgvector and chroma.

To capture PGVector and Chroma vectorstores request and response for in snapshot exit call detail, set the enable-genai-data-capture flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].