Debug distroless image using sidecar

Debug the Splunk Operator distroless image by attaching a sidecar container.

Since distroless images do not contain a shell, debugging requires a sidecar container that includes a shell and necessary utilities to inspect mapped volumes and files.

  1. Add a sidecar container to the splunk-operator-controller-manager deployment. The sidecar container has a shell and basic debugging tools.

    The following example deployment snippet adds a sidecar named sok-debug:

    CODE 
    apiVersion: apps/v1
kind: Deployment
metadata:
  name: splunk-operator-controller-manager
  namespace: splunk-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
        - name: manager
          image: splunk/splunk-operator:<version>-distroless
          env:
            - name: WATCH_NAMESPACE
              value: ""
            - name: RELATED_IMAGE_SPLUNK_ENTERPRISE
              value: splunk/splunk:<version>
            - name: SPLUNK_GENERAL_TERMS
              value: ""
        - name: sok-debug
          image: ubuntu:20.04
          command: ["/bin/bash", "-c", "tail -f /dev/null"]
          volumeMounts:
            - name: app-staging
              mountPath: /opt/splunk/appframework/
      volumes:
        - name: app-staging
          persistentVolumeClaim:
           claimName: splunk-operator-app-download
  2. Access the sidecar container by running an exec command into it:
    CODE 
    kubectl exec -it splunk-operator-pod-name -c sok-debug -- /bin/bash
  3. Navigate to the mounted volume to inspect files shared with the distroless container:
    CODE 
    cd /opt/splunk/appframework/
ls -l
  4. Check logs by accessing the appropriate log files or using kubectl logs:
    CODE 
    kubectl logs splunk-operator-pod-name -c manager
  5. After debugging, remove the sidecar container by editing the deployment and deleting the sidecar configuration, or redeploy the Splunk Operator without the sidecar.