Configure the Prometheus receiver to collect NVIDIA NIM metrics

Learn how to configure the Prometheus receiver to collect NVIDIA NIM metrics.

You can monitor the performance of NVIDIA NIMs by configuring your Kubernetes cluster to send NVIDIA NIM metrics to Splunk Observability Cloud.

This solution uses the Prometheus receiver to collect metrics from NVIDIA NIM, which can be installed on its own or as part of the NVIDIA NIM Operator. For more information on the NVIDIA NIM Operator, see About the Operator in the NVIDIA documentation. NVIDIA NIM exposes a :8000/metrics endpoint that publishes Prometheus-compatible metrics.

Complete the following steps to collect metrics from NVIDIA NIMs.

To use the Prometheus receiver to collect metrics from NVIDIA NIMs, you must meet the following requirements.

  • You have installed NVIDIA NIM using one of the following methods:

  • You have installed Prometheus for scraping metrics from NVIDIA NIM. For instructions, see Prometheus in the NVIDIA NIM documentation.

  1. Install the Splunk Distribution of the OpenTelemetry Collector for Kubernetes using Helm.
  2. To activate the Prometheus receiver for NVIDIA NIM manually in the Collector configuration, make the following changes to your configuration file:
    1. Add receiver_creator/nvidia to the receivers section. For more information on using the receiver creator receiver, see Receiver creator receiver.
    2. Add the Prometheus receiver for each NVIDIA NIM LLM that you want to monitor to the receiver_creator/nvidia section. For example, you can add prometheus/nim-llm, prometheus/nim-embedding, and/or prometheus/nim-reranking.
    3. Add receiver_creator/nvidia to the metrics pipeline of the service section.
      Example configuration file:
      YAML 
      agent:
  config:
    receivers:
      receiver_creator/nvidia:
        # Name of the extensions to watch for endpoints to start and stop.
        watch_observers: [ k8s_observer ]
        receivers:
          prometheus/nim-llm:
            config:
              config:
                scrape_configs:
                  - job_name: nim-for-llm-metrics
                    scrape_interval: 10s
                    metrics_path: /v1/metrics
                    static_configs:
                      - targets:
                          - '`endpoint`:8000'
            rule: type == "pod" && labels["app"] == "llm"

          prometheus/nim-embedding:
            config:
              config:
                scrape_configs:
                  - job_name: nim-for-embedqallm-metrics
                    scrape_interval: 10s
                    metrics_path: /v1/metrics
                    static_configs:
                      - targets:
                          - '`endpoint`:8000'
            rule: type == "pod" && labels["app"] == "embedqa"

          prometheus/nim-reranking:
            config:
              config:
                scrape_configs:
                  - job_name: nim-for-rerankqallm-metrics
                    scrape_interval: 10s
                    metrics_path: /v1/metrics
                    static_configs:
                      - targets:
                          - '`endpoint`:8000'
            rule: type == "pod" && labels["app"] == "rerankqa"
   service:
      pipelines:
        metrics/nvidianim-metrics:
          exporters:
            - signalfx
          processors:
            - memory_limiter
            - batch
            - resourcedetection
            - resource
          receivers:
            - receiver_creator/nvidia
  3. Restart the Splunk Distribution of the OpenTelemetry Collector.

Configuration settings

Learn about the configuration options for the Prometheus receiver.

To view the configuration options for the Prometheus receiver, see Settings.

Metrics

Learn about the available metrics for NVIDIA NIM.

For more information on the metrics available for NVIDIA NIM, see Observability for NVIDIA NIM for LLMs in the NVIDIA documentation.

These metrics are considered custom metrics in Splunk Observability Cloud.

Attributes

Learn about the available resource attributes are available for NVIDIA NIM.

The following resource attributes are available for NVIDIA NIM.
Resource attribute name Type Description Example value
model_name string The name of the deployed model.

meta/llama-3.1-8b-instruct
computationId string The unique identifier for the computation. comp-5678xyz