Monitor OpenAI APIs

AppDynamics Python Agent helps you monitor the OpenAI API usage and related metrics. It monitors the ChatCompletion, Completion, and Moderation APIs type requests created using the openai-python library to the OpenAI backend. The agent also captures the prompts and responses used by the OpenAI API models.

The following metrics are reported at the application, tier, and node levels:

Calls per minute
Errors per minute
Tokens
Input Tokens
Completion Tokens

The following are the example metric paths:

Per model metric path - Application Infrastructure Performance|OpenAI_Tier|OpenAI|ada|Tokens
All model cumulative metrics path - Application Infrastructure Performance|OpenAI_Tier|OpenAI|Tokens - All Models
Net application level metrics path - Overall Application Summary|OpenAI - Tokens - ada

Note: These metrics are reported even if the exit call is outside the business transaction. However, the Average Response Time(ms) metric is not reported when the exit call is outside the business transaction.

OpenAI Moderation API

From version 24.2.0, Python Agent supports OpenAI Moderation API. The moderation API will prompt the users if the queried statement is policy compliant or not. The Python Agent captures the following metrics:

Calls to moderations API: The number of times moderation API is used every minute. (OpenAI|moderations|Calls per minute)
Flagged category metrics: The number of times specific types of content, like hate speech, are flagged. (OpenAI|moderations|Flagged Calls by category|hate)
Flagged queries count: The total amount of flagged content. (OpenAI|moderations|Flagged queries)
Total queries count: The overall number of queries processed. (OpenAI|moderations|Total queries)

Prerequisites

Ensure that you meet the following prerequisites to monitor the OpenAI API exit calls with AppDynamics:

Install Python Agent 23.8.0 or later. See Install the Python Agent.
Import custom dashboards and templates.
Set the enable-openai flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].
To capture OpenAI request and response in snapshot exit call detail, set the enable-genai-data-capture flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].

Create a Custom Dashboard

Download Custom_dashboard_script.zip.

It contains:


Directory	Description
`templates`	Contains `json` templates to create the dashboard files.
`dashboard`	Dashboard `json` files are created here. It contains the sample `json` files.
`main.py`	Python script to create the dashboards.
`readme.md`	Description of the solution.

Run the following command:
```
cd Custom_dashboard_script
python3 main.py
```
The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.

Based on your selected dashboard type, you will be prompted to specify the following dashboard details.

Application Level Dashboard Details


Variable	Description	Default	Possible Values	Required
Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	`OpenAI_App_dashboard`	`app_dashboard`	Optional
Application Name	Name of the Application with OpenAI backend calls.	none	`openai_app_1`	Yes
OpenAI Endpoints Host Name	The hostname of the OpenAI endpoint.	`api.openai.com`	`api.openai.com`	Optional
OpenAI Endpoints Port	The port of the OpenAI endpoint.	`443`	`443`	Optional

Tier Level Dashboard Details


Variable	Description	Default	Possible Values	Required
Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	`tier_dashboard`	`tier_dashboard`	Optional
Application Name	Name of the Application with OpenAI backend calls.	none	`openai_app_1`	Yes
Tier Name	Name of the Tier exposed to OpenAI backend calls.	none	`openai_tier_1`	Yes
Business Transaction Name	The name of the business transaction reported to the host. This is to capture API Response time metric.	`none`	`openai_bt`	Yes
OpenAI Endpoints Host Name	The hostname of the OpenAI endpoint.	`api.openai.com`	`api.openai.com`	Optional
OpenAI Endpoints Port	The port of the OpenAI endpoint.	`443`	`443`	Optional

Application Level Moderation API Dashboard Details


Variable	Description	Default	Possible Values	Required
Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	`OpenAI_Moderation_App`	`moderation_app_dashboard`	Optional
Application Name	Name of the Application with OpenAI backend calls.	none	`openai_app_1`	Yes

Tier Level Moderation API Dashboard Details


Variable	Description	Default	Possible Values	Required
Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	`OpenAI_Moderation_Tier`	`moderation_tier_dashboard`	Optional
Application Name	Name of the Application with OpenAI backend calls.	none	`openai_app_1`	Yes
Tier Name	Name of the Tier exposed to OpenAI backend calls.	none	`openai_tier_1`	Yes

Import the dashboard in the Controller UI.
See Create Custom Dashboard.

Example Dashboard

Note: Navigate to Tiers & Nodes > My Dashboards and create a dashboard for the selected application by copying and pasting widgets from the existing dashboard for easier navigation.

Create a Custom Cost Metric Expression

You can create a custom cost metric expression by multiplying the Input Tokens and Completion Tokens metrics by their respective costs.

For example, the cost per one million tokens for gpt-3.5-turbo-0125 is:

Input Tokens = 0.5$
Completion Tokens = 1.5$

In this scenario, you can create a custom expression for gpt-3.5-turbo-0125 to calculate the cost as follows:

({input Tokens}*0.5 + {Completion Tokens}*1.5)/1000000

Use this custom expression to create a pie chart for cost distribution across different models or an overall cost widget that includes the cost function for all models. Complete the following steps to create a cost per model widget to the existing OpenAI Dashboard:

Add a pie chart widget to an existing openai dashboard, see Metric Expression.
Specify the input_token and completion_token cost and multiply them by the token count.
1. Data Source: Select the application name.
2. Metric Category: Select Tier / Node Health -Hardware, JVM, CLR (cpu,heap, disk I/O, etc).
3. Metric: Select Metric Expression.
4. Edit the metric Expression.
In Metric Expression, add two variables and assign names to them.
In Metric Selection, specify the relative metric path for input and completion tokens:
- Input Tokens: Agent|OpenAl|gpt-3.5-turbo|Input Tokens
- Completion Tokens: Agent|OpenAI|gpt-3.5-turbo|Completion Tokens

In Metric Expression, define the metric expression as follows:

({input_token_gpt3.5)*0.5 + {completion_token_gpt3.5)*1.5)/1000000

This metric expression includes the cost multiplier per million tokens for both types.

Documentation