Monitor Kubernetes

Learn how to monitor Kubernetes resources with Splunk Observability Cloud.

Note: This page describes features specific to Kubernetes navigators. For information on features shared by all navigators, see Use navigators in Splunk Infrastructure Monitoring.

Monitor and troubleshoot Kubernetes instances with Splunk Observability Cloud navigators. Splunk Observability Cloud uses the Splunk Distribution of OpenTelemetry Collector for Kubernetes to provide robust infrastructure monitoring capabilities. To learn more, see Get started with the Splunk Distribution of the OpenTelemetry Collector.

Use Kubernetes navigators to:

Get an overview of your Kubernetes infrastructure.
Monitor the health of your Kubernetes infrastructure.
Identify and diagnose an issue with your Kubernetes infrastructure.
View services and hosts running on Kubernetes.

For the full list of supported Kubernetes navigators, see Available navigators.

Prerequisites

To start monitoring Kubernetes resources, you must:

Collect Kubernetes data.
Log in with your administrator credentials.

Access Kubernetes navigators

To access Kubernetes navigators:

In the Splunk Observability Cloud main menu, select Infrastructure.
Select Kubernetes. The Kubernetes section displays the summary cards for Kubernetes navigators.
Select a summary card to access a Kubernetes navigator.

Feature availability for Kubernetes navigators

The following table lists the features specific to Kubernetes navigators, describes where each feature is available, and links to the documentation on this page for each feature.


Feature name	Available on these Kubernetes navigators	Link to documentation
Left navigation panel	All	Refine your view with the left navigation panel
Hierarchy map K8s analyzer tab	Nodes, pods, containers	Investigate instances with the hierarchy map Troubleshoot performance with the Kubernetes analyzer
Overview tab K8s events tab	Nodes, pods	Monitor entity performance with the Overview tab View Kubernetes events associated with an instance
Search embedded logs	Nodes	Search embedded logs

Use the left navigation panel to quickly switch between Kubernetes entity types, search for filters, use predefined filters, and view or use recently used filters. This panel is available in all Kubernetes navigators and views.

To refine your view with the left navigation panel, use the following features:

Select entity type: Use this drop-down menu to switch between Kubernetes entity types.
Refine by: Use this panel to search for filters, use predefined filters, or view and use recently used filters. The list of predefined filters is searchable and organized by Relationship and Attribute.

Monitor entity performance with the Overview tab

Note: The Overview tab is only available on the Kubernetes nodes and pods navigators.

The Overview tab displays by default when you select the Kubernetes nodes or pods navigators. This view includes a dashboard that summarizes the performance of the selected entity type. Use this view to quickly identify the instances that consume the most resources, cause the highest number of errors, and contribute to slowing down your infrastructure.

This view also includes the K8s analyzer, K8s events, and Dependencies tabs.

Investigate instances with the hierarchy map

Note: The hierarchy map is only available on the detail view of the Kubernetes nodes, pods, and containers navigators.

Monitor your Kubernetes infrastructure with an interactive hierarchical map that displays the child resources associated with a selected Kubernetes instance. You can select elements in the map to drill down into them, or use the filter to explore your data. The level of detail shown on the map is dynamic and depends on the number of elements shown.

To navigate to the hierarchy map:

In the Splunk Observability Cloud main menu, select Infrastructure, then Kubernetes.
Select the Kubernetes nodes, pods, or containers navigator.
Navigate to the table view:
1. If you selected the Kubernetes nodes or pods navigator, select the Table tab.
2. If you selected the Kubernetes containers navigator, no action is required. The table view displays by default.
In the table, select an instance name to navigate to the detail view.
Expand the Hierarchy Map.

Hierarchy map features

On the hierarchy map, nodes, pods, and containers are colored by health and status, as reported by Kubernetes:

Nodes are colored by condition: Node Ready, Memory Pressure, PID Pressure, Disk Pressure, Network Unavailable, and Out of Disk
Pods are colored by phase: Running, Pending, Succeeded, Failed, and Unknown
Containers are colored by status: Ready, Not Ready, and Unknown

To investigate instances with the hierarchy map, use the following features:

Breadcrumb navigation: Switch to different instances and jump across entity levels using the breadcrumb navigation bar.
Hover: Get more information about an instance, including its status or phase, by hovering over that instance.
Select and zoom: Drill down into an instance and change the zoom level of the map, if applicable, by selecting the instance.
Filter: Filter the map by any available metadata in your Kubernetes data, such as a namespace, a workload, or any other key-value pair. When you apply a filter, the map highlights instances that match the filter. You can still hover over the dimmed instances to view details about them.

View Kubernetes events associated with an instance

Note: The K8s events tab is only available on the overview and detail views of the Kubernetes nodes and pods navigators. This feature requires enabling Log Observer Connect.

Select the K8s events tab to view an events rate chart that groups events by severity and a searchable table that lists the Kubernetes events associated with the entity type.

Search embedded logs

Note: This feature is only available on the overview and detail view of the Kubernetes nodes navigator.

Search for specific keywords within logs embedded in Kubernetes navigators for faster troubleshooting and log analysis. Your search does not affect the Log Chart Summary, ensuring data integrity.

To search embedded logs:

Select the K8s node logs tab.
In the search bar next to the System logs header or the Authentication logs header, search for the keyword that you want to find in embedded logs.
Note: Searches are case insensitive and treat the keywords you enter as a single string, aligning with Log Observer Connect behavior. When you view the logs in Log Observer Connect, the search persists to maintain context.

Troubleshoot performance with the Kubernetes analyzer

Note: The Kubernetes analyzer is only available on the:

Overview of the Kubernetes nodes and pods navigators.
Table or heat map view of the Kubernetes containers navigator.

Select the K8s analyzer tab to access the Kubernetes analyzer. The analyzer helps you troubleshoot Kubernetes problems at scale by highlighting Kubernetes instances that are in a bad state, such as nodes that are not ready. The analyzer produces theories about what those instances might have in common, such as that all of the instances are running the same workload or all instances are located in the same AWS region. Select a finding in the analyzer to filter the map.

The analyzer displays suggested filters for the elements selected in the table or heat map view. Select links in the analyzer to add filters to the table or heat map view and explore conditions across your entire Kubernetes environment.

The analyzer uses AI-driven insights to examine potential patterns between nodes, pods, or containers. The trouble indicators are:

Pods that are in pending status
Pods that are in failed status
Pods with unknown condition
Containers with high restart counts
Nodes not ready
Nodes with unknown condition
Nodes experiencing high CPU
Nodes experiencing high memory

The analyzer displays overrepresented metrics properties for known conditions, such as pods in pending status, pods in failed status, and so on. You can use properties that are highly correlated with these conditions to filter the table or heat map.

You can also explore data about each of those elements in the navigator using context-sensitive dashboards. This enables you to identify the underlying patterns noticeable on the filtered map that might be correlated with Kubernetes issues. For example, if all failed pods are in certain types of clusters, the analyzer provides suggested paths to follow to troubleshoot such issues.

Documentation