Kubernetes data collectors

Collect Kubernetes cluster data for troubleshooting and analysis.

The Splunk Operator for Kubernetes deploys Splunk Enterprise custom resources across one or more namespaces. Helper scripts in the Splunk Operator repository collect cluster data for troubleshooting and analysis.

The Splunk Operator for Kubernetes deploys Splunk Enterprise custom resources across a single namespace or multiple namespaces. The helper scripts k8s-splunk-collector.sh and k8s-systeminfo-collector.sh in the tools directory on GitHub collect data from a Kubernetes cluster that runs the Splunk Operator for Kubernetes.

Splunk collector output layout and performance

Folder layout for data collected by k8s-splunk-collector.sh and sample performance metrics from internal testing.

Target folder breakdown

<target_folder>/k8s_data/get
Stores output from kubectl get commands for Kubernetes resources (deployments, StatefulSets, ConfigMaps, services, volumes, RBAC), including Splunk Enterprise custom resources in the cluster.
<target_folder>/k8s_data/describe
Stores output from kubectl describe commands for those resources. The script skips this folder when you avoid describe commands.
<target_folder>/pods_data/diags
Stores diags from all Splunk instances running inside Splunk Enterprise custom resource pods that the Splunk Operator for Kubernetes deploys. The script creates this folder only when you opt in to diags.
<target_folder>/pod_data/logs
Stores logs from all Splunk Enterprise custom resource pods and from the Operator pod.

Logs use clear naming and prefixes. The k8s_data folder contains kubectl get and describe output for multiple resources. The pods_data folder contains pod logs and, when requested, diags from Splunk pods.

Performance

Example Splunk deployments on a Kubernetes cluster: Search Head Cluster (3 search heads, 1 deployer), Indexer Cluster (3 indexers, 1 cluster manager), 1 standalone, 1 license manager.

CODE 
kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
splunk-default-monitoring-console-0   1/1     Running   0          17m
splunk-example-license-manager-0       1/1     Running   0          18m
splunk-operator-cb8d66765-tl6z2       1/1     Running   0          6h6m
splunk-test-cluster-manager-0          1/1     Running   0          19m
splunk-test-deployer-0                1/1     Running   0          6h3m
splunk-test-indexer-0                 1/1     Running   0          17m
splunk-test-indexer-1                 1/1     Running   0          17m
splunk-test-indexer-2                 1/1     Running   0          17m
splunk-test-search-head-0             1/1     Running   0          6h3m
splunk-test-search-head-1             1/1     Running   0          6h3m
splunk-test-search-head-2             1/1     Running   0          6h3m
splunk-test2-standalone-0             1/1     Running   0          6h5m

For performance testing, the script:

  • Collected diags (-d set to true)
  • Collected kubectl describe commands (-l not supplied)
  • Collected secret metadata (-s set to true)

Performance metrics:

CODE 
Time taken - 6 mins 40 seconds
Memory - 774.4 MB

Run the Splunk collector script (k8s-splunk-collector.sh)

Collect kubectl command output, container logs, and optional Splunk diags from Splunk Enterprise pods in the current namespace using k8s-splunk-collector.sh. The Splunk collector for Kubernetes gathers data from multiple kubectl get and describe commands, container logs, and optional diags from Splunk instances in the current namespace.

The script:

  • Collects extensive data on your Kubernetes cluster. If you must keep any data private, do not run the script as-is; modify it to suit your needs.
  • If you opt in to diags, generates a Splunk diag on each Splunk instance running in Splunk Enterprise custom resource pods that the Splunk Operator for Kubernetes deploys. The script removes the diag from the Splunk instance after extraction.

If you do not want either behavior, collect data manually instead.

Prerequisites

  • Kubeconfig context set to the cluster and namespace where the Splunk Operator for Kubernetes runs
  • Access to run kubectl commands to gather data
  • Permission to create and delete directories on the host file system (at least in the directory where you run the script and store data)
  • Enough disk space for the target folder. See the Performance section for sizing.

Steps

  1. Run the script with the following syntax:
    CODE 
    sh k8s-splunk-collector.sh -d <flag_to_collect_splunk_diags> -t <target_folder> -l <flag_to_limit_output_by_avoid_kubectl_describe> -s <flag_to_collect_secret_object_metadata>

    Configurable options:

    • -d. Whether to collect Splunk diags. By default, the script does not collect Splunk diags. Set to true to collect them.
    • -t. Target folder. Optional. Without -t, the script creates a timestamped folder tmp-<timestamp> in the current working directory. With -t and a valid full path, it creates tmp-<timestamp> inside that path. If the path does not exist, the script creates it when at least one parent path exists; if no parent path exists, the script finishes without writing data. Ensure enough free space in either case.
    • -l. Set to true to skip kubectl describe output and limit collection. By default the script collects describe output. A known Kubernetes issue can create many clients for describe commands (Kubernetes issue 91913). Internal testing did not show functional problems from the warnings, but you can set -l to true to reduce warnings and protect limited network bandwidth.
    • -s. Whether to collect Kubernetes secret object metadata. Off by default. Set to true to collect metadata only. The script does not collect sensitive secret data.
  2. Wait until the script prints All data required collected under folder <target_folder>.

Example run

CODE 
sh k8s-splunk-collector.sh -d "true"
Starting to collect data with diag true in folder /Users/akondur/Desktop/operator_training/Data_collection_debug/collect_data_k8s/tmp-2021-04-19-10-37 

Setting up directories 

Done setting up directories 

Started collecting logs and diags

Done collecting logs and diags

Started collecting cluster info

Done collecting cluster info

Started collecting kubectl get command outputs

Done collecting kubectl get command outputs

Started collecting kubectl describe command outputs

Done collecting kubectl describe command outputs 

All data required collected under folder /Users/akondur/Desktop/operator_training/Data_collection_debug/collect_data_k8s/tmp-2021-04-19-10-37

Run the system info collector script (k8s-systeminfo-collector.sh)

Collect Kubernetes-related system information from a cluster node using k8s-systeminfo-collector.sh. Only a Kubernetes administrator should run this script. The system info collector gathers Kubernetes-related system information that Splunk support typically needs.

Prerequisites

  • Run the script on a Kubernetes node (SSH access to the node). This topic does not document SSH setup.
  • Administrator access to the Kubernetes cluster

Steps

  1. Run the script:
    CODE 
    sh k8s-systeminfo-collector.sh --ignore_introspection <ignore_introspection_flag> --ignore_metrics <ignore_metrics_flag>

    Options:

    • ignore_introspection. Set to true to skip introspection data. By default the script collects introspection data.
    • ignore_metrics. Set to true to skip metrics data. By default the script collects metrics data.

Example output

JAVASCRIPT 
sh-4.2$ sudo tools/k8s-log-collector.sh

        Version 0.0.1. For updates see https://github.com/splunk/splunk-operator/tools

Trying to collect common operating system logs...
Trying to collect kernel logs...
Trying to collect mount points and volume information...
Trying to collect SELinux status...
Trying to collect iptables information...
Trying to collect installed packages...
Trying to collect active system services...
Trying to Collect Containerd daemon information...
Trying to collect Docker daemon information...
Trying to collect kubelet information...
Trying to collect L-IPAMD introspection information... Trying to collect L-IPAMD prometheus metrics... Trying to collect L-IPAMD checkpoint... cp: cannot stat '/var/run/k8s-node/ipam.json': No such file or directory

Trying to collect sysctls information...
Trying to collect networking infomation... conntrack v1.4.4 (conntrack-tools): 253 flow entries shown.

Trying to collect CNI configuration information...
Trying to collect Docker daemon logs...
Trying to archive gathered information...

        Done... bundled logs: /var/log/k8s__2022-03-10_1857-UTC_0.0.1.tar.gz
Note: The example uses k8s-log-collector.sh in the sample transcript; align the executable name with your Splunk Operator tools package and the script described in this topic (k8s-systeminfo-collector.sh).

Output contents

The script writes a tar file in the working directory from which you run it. After you extract the archive, you can find content such as:

  1. Kernel logs under <target_folder>/kernel
  2. Mount points and volume information under <target_folder>/storage
  3. SELinux status under <target_folder>/system
  4. IPtables under <target_folder>/networking
  5. Installed packages under <target_folder>/system
  6. System services under <target_folder>/system
  7. Containerd under <target_folder>/containerd
  8. Docker under <target_folder>/docker
  9. Kubelet under <target_folder>/kubelet
  10. IPAMD under <target_folder>/ipamd
  11. Sysctls under <target_folder>/sysctls
  12. Networking (conntrack, ifconfig, routes, and similar) under <target_folder>/networking
  13. CNI under <target_folder>/cni
  14. Docker logs for the system under <target_folder>/var_log

Example extracted layout

CODE 
drwxr-xr-x 4 root root   4096 Mar 16 22:26 var_log
drwxr-xr-x 2 root root    137 Mar 16 22:26 system
drwxr-xr-x 2 root root     86 Mar 16 22:26 storage
drwxr-xr-x 2 root root     89 Mar 16 22:26 kernel
drwxr-xr-x 2 root root     61 Mar 16 22:26 containerd
drwxr-xr-x 2 root root     28 Mar 16 22:26 sysctls
drwxr-xr-x 2 root root    249 Mar 16 22:26 networking
drwxr-xr-x 2 root root     75 Mar 16 22:26 kubelet
drwxr-xr-x 2 root root    153 Mar 16 22:26 ipamd
drwxr-xr-x 2 root root    143 Mar 16 22:26 docker
drwxr-xr-x 2 root root     29 Mar 16 22:26 cni