Configure your Splunk Observability Cloud account to collect Azure OpenAI metrics

Learn how to configure your Splunk Observability Cloud account to collect Azure OpenAI metrics.

You can monitor the performance of Azure OpenAI applications by configuring your Azure OpenAI applications to send metrics to Splunk Observability Cloud. This solution creates a cloud connection in your Splunk Observability Cloud account that collects metrics from Azure application insights.

Complete the following steps to collect metrics from your Azure OpenAI applications.

  1. Connect Azure to Splunk Observability Cloud. For more information on the connection methods and instructions for each method, see Available options to connect with Azure.
    Note: If you connect with Azure using the guided setup, the Select the data to import page includes the option to import data from All Azure services or Specified Azure services only. If you select Specified Azure services only, select Cognitive Services from the Specified services drop-down menu to ensure that Azure OpenAI metrics are ingested.
  2. To monitor Azure OpenAI metrics with Splunk Observability Cloud, run your applications that use Azure OpenAI models.

Metrics and attributes

Learn about the available metrics for Azure OpenAI.

The following metrics and resource attributes are available for Azure OpenAI applications. These metrics fall under the default metric category.

Metric nameTypeUnitDescriptionDimensions
ProcessedPromptTokenscountercountThe number of prompt tokens processed (input) on an OpenAI model.

  • ApiName

  • ModelDeploymentName

  • FeatureName

  • UsageChannel

  • Region

  • ModelVersion

GeneratedTokens

counter

countThe number of tokens generated (output) on an OpenAI model.

  • ApiName

  • ModelDeploymentName

  • FeatureName

  • UsageChannel

  • Region

  • ModelVersion

AzureOpenAIRequestscountercountThe number of calls made to the Azure OpenAI service.

  • ApiName

  • OperationName

  • Region

  • ModelDeploymentName

  • ModelName

  • ModelVersion

  • StatusCode

AzureOpenAITimeToResponsehistogrammsThe time taken for the first response to appear after the user sends a prompt.

  • ApiName

  • OperationName

  • Region

  • ModelDeploymentName

  • ModelName

  • ModelVersion

  • StatusCode

AzureOpenAIAvailabilityRategaugepercentThe availability percentage with the following calculation: (Total Calls - Server Errors)/Total Calls. Server Errors include any HTTP responses >=500.

  • ApiName

  • OperationName

  • Region

  • ModelDeploymentName

  • ModelName

  • ModelVersion

  • StatusCode

AzureOpenAITokenPerSecond

gaugecountThe generation speed for a given Azure OpenAI model response.

  • Region

  • ModelDeploymentName

  • ModelName

  • ModelVersion

AzureOpenAIContextTokensCacheMatchRategaugepercentThe percentage of prompt tokens that hit the cache.

  • Region

  • ModelDeploymentName

  • ModelName

  • ModelVersion

Next steps

How to monitor your AI components after you set up Observability for AI.

After you set up data collection from supported AI components to Splunk Observability Cloud, the data populates built-in experiences that you can use to monitor and troubleshoot your AI components.

The following table describes the tools you can use to monitor and troubleshoot your AI components.
Monitoring toolUse this tool toLink to documentation
Built-in navigatorsOrient and explore different layers of your AI tech stack.
Built-in dashboardsAssess service, endpoint, and system health at a glance.
Splunk Application Performance Monitoring (APM) service map and trace viewView all of your LLM service dependency graphs and user interactions in the service map or trace view.

Monitor LLM services with Splunk APM