Configure the Prometheus receiver to collect NVIDIA GPU metrics
Learn how to configure the Prometheus receiver to collect NVIDIA GPU metrics.
You can monitor the performance of NVIDIA GPUs by configuring your Kubernetes cluster to send NVIDIA GPU metrics to Splunk Observability Cloud. This solution uses the Prometheus receiver to collect metrics from the NVIDIA DCGM Exporter, which can be installed independently or as part of the NVIDIA GPU Operator.
For more information on these NVIDIA components, see the NVIDIA DCGM Exporter GitHub repository and About the NVIDIA GPU Operator in the NVIDIA documentation. The NVIDIA DCGM Exporter exposes a /metrics endpoint that publishes Prometheus-compatible metrics.
Complete the following steps to collect metrics from and monitor the performance of NVIDIA GPUs.
Prerequisite
Learn how to install the NVIDIA DCGM Exporter on your Kubernetes cluster, which is a prerequisite to configure the Prometheus receiver to collect NVIDIA GPU metrics.
To configure the Prometheus receiver to collect metrics from NVIDIA GPUs, you must install the NVIDIA DCGM Exporter on your Kubernetes cluster. You can use one of the following methods:
To install the NVIDIA DCGM Exporter using Helm, see Quickstart on Kubernetes in the NVIDIA DCGM Exporter GitHub repository.
- To install the NVIDIA DCGM Exporter as part of the NVIDIA GPU Operator, see Installing the NVIDIA GPU Operator in the NVIDIA documentation.
Configure and activate the component for NVIDIA GPU
Learn how to configure and activate the component for Nvidia GPU.
Complete the following steps to configure and activate the component for NVIDIA GPU.
Monitor the performance of NVIDIA GPUs
Learn how to navigate to the Nvidia GPU navigator, which you can use to monitor the performance of NVIDIA GPUs.
Complete the following steps to access the NVIDIA GPU navigator and monitor the performance of NVIDIA GPUs. For more information on navigators, see Use navigators.
- From the Splunk Observability Cloud main menu, select Infrastructure.
- Under AI/ML, select AI Frameworks.
- Select the NVIDIA GPU summary card.
Configuration settings
Learn about the configuration settings for the Prometheus receiver.
To view the configuration options for the Prometheus receiver, see Settings.
Metrics
Learn about the available monitoring metrics for Nvidia GPUs.
Metric name | Type | Unit | Description |
---|---|---|---|
DCGM_FI_DEV_SM_CLOCK | gauge | MHz | SM clock frequency. |
DCGM_FI_DEV_MEM_CLOCK | gauge | MHz | Memory clock frequency. |
DCGM_FI_DEV_MEMORY_TEMP | gauge | C | Memory temperature. |
DCGM_FI_DEV_GPU_TEMP | gauge | C | GPU temperature. |
DCGM_FI_DEV_POWER_USAGE | gauge | W | Power draw. |
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | counter | mJ | Total energy consumption since boot. |
DCGM_FI_DEV_PCIE_REPLAY_COUNTER | counter | count | Total number of PCIe retries. |
DCGM_FI_DEV_GPU_UTIL | gauge | percent | GPU utilization. |
DCGM_FI_DEV_MEM_COPY_UTIL | gauge | percent | Memory utilization. |
DCGM_FI_DEV_ENC_UTIL | gauge | percent | Encoder utilization. |
DCGM_FI_DEV_DEC_UTIL | gauge | percent | Decoder utilization. |
DCGM_FI_DEV_FB_FREE | gauge | MiB | Framebuffer memory free. |
DCGM_FI_DEV_FB_USED | gauge | MiB | Framebuffer memory used. |
DCGM_FI_PROF_PCIE_TX_BYTES | counter | bytes | Number of bytes of active PCIe TX data, including both header and payload. |
DCGM_FI_PROF_PCIE_RX_BYTES | counter | bytes | Number of bytes of active PCIe RX data, including both header and payload. |
Attributes
Learn about the available attributes for NVIDIA GPUs.
The following attributes are available for NVIDIA GPUs.
Attribute name | Type | Description | Example value |
---|---|---|---|
app | string | The name of the application attached to the GPU. | nvidia-dcgm-exporter |
DCGM_FI_DRIVER_VERSION | string | The version of the NVIDIA DCGM driver installed on the system. | 570.124.06 |
device | string | The identifier for the specific NVIDIA device or GPU instance. | nvidia0 |
gpu | number | The index number of the GPU within the system. | 0 |
modelName | string | The commercial model of the NVIDIA GPU. | NVIDIA A10G |
UUID | string | A unique identifier assigned to the GPU. | GPU-3ca2f6af-10d6-30a5-b45b-158fc83e6d33 |
Troubleshoot
Learn how to get help if you can't see your data in Splunk Observability Cloud.
If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways:
Splunk Observability Cloud customers can submit a case in the Splunk Support Portal or contact Splunk Support.
Prospective customers and free trial users can ask a question and get answers through community support in the Splunk Community.