Monitor Kubernetes

Learn how to monitor Kubernetes resources with Splunk Observability Cloud.

You can monitor Kubernetes metrics with Splunk Observability Cloud navigators. Splunk Observability Cloud uses the Splunk Distribution of OpenTelemetry Collector for Kubernetes to provide robust infrastructure monitoring capabilities. To learn more, see Get started with the Splunk Distribution of the OpenTelemetry Collector.

Using Kubernetes navigators, you can:

  • Get an overview of your Kubernetes infrastructure.

  • Monitor the health of your Kubernetes infrastructure.

  • Identify and diagnose an issue with your Kubernetes infrastructure.

  • View services and hosts running on Kubernetes.

Prerequisites

To start monitoring Kubernetes resources, you must:

Access Kubernetes navigators

Note: The following sections describe components specific to the Kubernetes navigators. For information on components shared by all navigators, see Use navigators in Splunk Infrastructure Monitoring.

To access Kubernetes navigators, select Infrastructure from the Splunk Observability Cloud main menu. The Kubernetes section displays the summary cards for Kubernetes navigators. Select a summary card to access a Kubernetes navigator.

Splunk Observability Cloud supports the following Kubernetes navigators:

  • Clusters

  • Nodes

  • Pods

  • Containers

  • Workloads

  • Deployments

  • ReplicaSets

  • StatefulSets

  • DaemonSets

  • Jobs

  • CronJobs

  • Services

  • Resources

Investigate instances with the hierarchy map

Note: The hierarchy map is only available on the Kubernetes nodes, pods, and containers navigators.

Monitor your Kubernetes infrastructure with an interactive hierarchical map that displays the child resources associated with a selected Kubernetes instance. You can select elements in the map to drill down into them, or use the filter to explore your data. The level of detail shown on the map is dynamic and depends on the number of elements shown.

To navigate to the hierarchy map:

  1. From the Splunk Observability Cloud main menu, select Infrastructure, then Kubernetes.

  2. Select the Kubernetes nodes, pods, or containers navigator.

  3. The table view displays by default. Select an instance from the table.

  4. Expand the Hierarchy Map.

Nodes, pods, and containers are colored by health and status, as reported by Kubernetes:

  • Nodes are colored by condition: Node Ready, Memory Pressure, PID Pressure, Disk Pressure, Network Unavailable, and Out of Disk

  • Pods are colored by phase: Running, Pending, Succeeded, Failed, and Unknown

  • Containers are colored by status: Ready, Not Ready, and Unknown

Hierarchy map features

To investigate instances with the hierarchy map, use the following features:

  • Breadcrumb navigation: Switch to different instances and jump across entity levels using the breadcrumb navigation bar.

  • Hover: Get more information about an instance, including its status or phase, by hovering over that instance.

  • Select and zoom: Drill down into an instance and change the zoom level of the map, if applicable, by selecting the instance.

  • Filter: Filter the map by any available metadata in your Kubernetes data, such as a namespace, a workload, or any other key-value pair. When you apply a filter, the map highlights instances that match the filter. You can still hover over the dimmed instances to view details about them.

Refine your view with the left navigation panel

Use the left navigation panel to quickly switch between Kubernetes entity types, search for filters, use predefined filters, and view or use recently used filters.

The left navigation panel is available in the aggregate views (table view and heat map view). For more information about these views and how to navigate to them, see Monitor all instances in a navigator.

To refine your view with the left navigation panel, use the following features:

  • Select entity type: Use this drop-down menu to switch between Kubernetes entity types.

  • Refine by: Use this panel to search for filters, use predefined filters, or view and use recently used filters. The list of predefined filters is searchable and organized by Relationship and Attribute.

Search embedded logs

You can search for specific keywords within logs embedded in Kubernetes navigators, dashboards, and in the APM service-centric view. Your search does not affect the Log Chart Summary, ensuring data integrity.

To search embedded logs in Kubernetes navigators, follow these steps:

  1. In Splunk Observability Cloud, select Infrastructure then navigate to a Kubernetes node in your environment. Next to the System logs header on the first table or Authentication logs header on the second table, enter the keyword in the search bar that you want to search for in embedded logs.

    Note: Searches are case-insensitive and treat the keywords you enter as a single string, aligning with Log Observer Connect behavior. When you view the logs in Log Observer Connect, the search persists to maintain context

  2. Press Enter on your keyboard. (There is no Search button.)

Troubleshoot performance with the analyzer

Note: The analyzer is only available on the Kubernetes nodes, pods, and containers navigators.

Select the K8s analyzer tab to access the Kubernetes analyzer. The analyzer helps you troubleshoot Kubernetes problems at scale by highlighting Kubernetes instances that are in a bad state, such as nodes that are not ready. The analyzer produces theories about what those instances might have in common, such as that all of the instances are running the same workload or all instances are located in the same AWS region. Select a finding in the analyzer to filter the map.

The analyzer displays suggested filters for the elements selected in the table or heat map view. Select links in the analyzer to add filters to the table or heat map view and explore conditions across your entire Kubernetes environment.

The analyzer uses AI-driven insights to examine potential patterns between nodes, pods, or containers. The trouble indicators are:

  • Pods that are in pending status

  • Pods that are in failed status

  • Pods with unknown condition

  • Containers with high restart counts

  • Nodes not ready

  • Nodes with unknown condition

  • Nodes experiencing high CPU

  • Nodes experiencing high memory

The analyzer displays overrepresented metrics properties for known conditions, such as pods in pending status, pods in failed status, and so on. You can use properties that are highly correlated with these conditions to filter the table or heat map. You can explore data about each of those elements in the navigator using context-sensitive dashboards. This enables you to identify the underlying patterns noticeable on the filtered map that might be correlated with Kubernetes issues. For example, if all failed pods are in certain types of clusters, the analyzer provides suggested paths to follow to troubleshoot such issues.

Next steps

You can also export and monitor data related to your Kubernetes clusters, as described in the following table.

Get data in

Monitor

Description

Connect to the cloud service provider your Kubernetes clusters run in, if any.

Instrument back-end applications to send spans to Splunk APM

Introduction to Splunk APM

Collect metrics and spans from applications running in Kubernetes clusters.