Troubleshoot Splunk custom resources

Diagnose issues with Splunk Operator custom resources using CR status messages, pause annotations, admin-managed persistent volumes, and container logs.

CR status message

The Splunk Enterprise CRDs with the Splunk Operator have a field cr.Status.message that provides a detailed view of the CR current status.

The following example shows a Standalone with a message indicating an invalid CR config:

bash% kubectl get stdaln
NAME   PHASE   DESIRED   READY   AGE   MESSAGE
ido    Error   0         0       26s   invalid Volume Name for App Source: custom. volume: csh, doesn't exist

bash# kubectl get stdaln -o yaml | grep -i message -A 5 -B 5
      appsStatusMaxConcurrentAppDownloads: 5
      bundlePushStatus: {}
      isDeploymentInProgress: false
      lastAppInfoCheckTime: 0
      version: 0
    message: 'invalid Volume Name for App Source: custom. volume: csh, doesn''t exist'
    phase: Error
    readyReplicas: 0
    replicas: 0
    resourceRevMap: {}
    selector: ""

Pause annotations

The Splunk Operator controller reconciles every Splunk Enterprise CR. However, you might need to pause the Splunk Operator reconciliation in certain circumstances. Every Splunk Enterprise CR has its own pause annotation. When you configure this annotation, the Splunk Operator controller pauses reconciliation for that CR.

The following example shows a Standalone with the pause annotation set. In this state, the Splunk Operator requeues the reconciliation without performing any reconcile operations unless the annotation is removed.

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: test-only-debug
  namespace: splunk-operator
  annotations:
    standalone.enterprise.splunk.com/paused: "true"
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 1

admin-managed-pv annotations

The admin-managed-pv annotation in the Splunk Operator Custom Resource allows the admin to control whether the operator dynamically creates Persistent Volumes (PVs) for the StatefulSet associated with the CR.

If set to true, the operator does not create PVs, and the Persistent Volume Claim templates in the StatefulSet manifest include a selector block to match app.kubernetes.io/instance and app.kubernetes.io/name labels for pre-created PVs. This means that /opt/splunk/etc and /opt/splunk/var related PVCs contain a code block like the following:

apiVersion: v1
kind: PersistentVolumeClaim
...
  selector:
    matchLabels:
      app.kubernetes.io/instance: splunk-cm-cluster-manager
      app.kubernetes.io/name: cluster-manager

To match this selector definition, the Persistent Volume must set labels accordingly:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-example-etc
  labels:
    app.kubernetes.io/instance: splunk-cm-cluster-manager
    app.kubernetes.io/name: cluster-manager

When admin-managed-pv is set to false, the operator dynamically creates PVs as usual, providing dedicated persistent storage for the StatefulSet.

The following example shows a Standalone with the admin-managed-pv annotation set:

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: single
  finalizers:
  - enterprise.splunk.com/delete-pvc
  annotations:
    enterprise.splunk.com/admin-managed-pv: "true"

PV label values

To prepare labels for CR persistent volumes, you need to know the values beforehand. The following table lists app.kubernetes.io/name values mapped to CRDs:

The app.kubernetes.io/instance value consists of three elements concatenated with hyphens:

• splunk
• The CR name provided by the admin
• The CRD kind name

For example, a clusterManager CR named "test" has the app.kubernetes.io/instance value splunk-test-cluster-manager.

Container logs

The Splunk Enterprise CRDs deploy Splunkd in Kubernetes pods running docker-splunk container images. Adding the following environment variables to the CR spec produces detailed container logs:

apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: test-only
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 1
  extraEnv:
  - name: DEBUG
    value: "true"
  - name: ANSIBLE_EXTRA_FLAGS
    value: "-vvvv"

The following snippet shows a sample from the detailed container log:

TASK [splunk_common : Ensure license path] *************************************
task path: /opt/ansible/roles/splunk_common/tasks/licenses/add_license.yml:15
ok: [localhost] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "checksum_algorithm": "sha1",
            "follow": false,
            "get_attributes": true,
            "get_checksum": true,
            "get_md5": false,
            "get_mime": true,
            "path": "splunk.lic"
        }
    },
    "stat": {
        "exists": false
    }
}