Troubleshoot Universal Forwarder deployment issues

Resolve pod startup failures, missing forwarding configuration, and Helm upgrade timeouts seen when deploying the Universal Forwarder chart.

Enable verbose Ansible logging for more detail while diagnosing any of the issues in this topic.

Tip: Upgrade the release with debug.enabled=true to turn on verbose Ansible output (SPLUNK_ANSIBLE_DEBUG=true), then check the container logs.
CODE 
helm upgrade my-uf ./helm-chart/splunk-universalforwarder \
  --namespace my-namespace \
  --values values-production.yaml \
  --set debug.enabled=true
CODE 
kubectl logs -n my-namespace <pod-name> -c splunk-universalforwarder | head -200

Pod stuck in Init:CrashLoopBackOff

Symptom: The pod remains in Init:CrashLoopBackOff. Check which init container is failing:

CODE 
kubectl describe pod <pod-name> -n my-namespace

Cause: fix-ownership is failing. This is usually a policy constraint. The pod events show:

CODE 
container's runAsUser breaks non-root policy

Your cluster has a PodSecurityAdmission policy that blocks runAsUser: 0.

Remedy:

  1. Relax the policy for the namespace.
  2. Alternatively, pre-provision the PVCs with the correct uid.

Cause: seed-etc is failing with Permission denied. The PVC was not chowned before seed-etc ran. This should not happen with the current chart.

Remedy: Verify that fix-ownership ran and exited 0 first.

Pod stuck in Pending

Symptom: The pod remains in Pending. Check the pod events:

CODE 
kubectl describe pod <pod-name> -n my-namespace | grep -A 10 Events

Cause: A PVC is still terminating from a previous deployment.

Remedy: Wait for the EBS volume to detach, or check whether old pods are still holding the PVC mount.

Cause: No nodes are available.

Remedy: Check node taints and tolerations.

Cause: The StorageClass does not exist.

Remedy: Verify the value of storage.storageClassName.

outputs.conf not written or forwarding not working

Symptom: Check outputs.conf and the splunkd log for connection attempts:

CODE 
# Check outputs.conf content
kubectl exec -n my-namespace <pod-name> -- \
  cat /opt/splunkforwarder/etc/system/local/outputs.conf

# Check splunkd log for connection attempts
kubectl exec -n my-namespace <pod-name> -- \
  grep -E "TcpOutput|AutoLoadBalanced|Connected|forward" \
  /opt/splunkforwarder/var/log/splunk/splunkd.log | tail -20

If outputs.conf only contains [indexAndForward] index = false with no [tcpout] block, the splunk add forward-server call failed.

Cause: Password mismatch: a stale PVC has a different password than the current Secret.

Remedy: Reconcile the password in the Secret with the password stored on the retained PVC, or rotate the password as described in Manage the Universal Forwarder admin password.

Cause: Both forward_servers and s2s_port must be set in default.yml. forward_servers alone does not trigger the CLI call.

Remedy: Confirm that both forward_servers and s2s_port are set, then reinstall or upgrade the release.

Helm upgrade fails with context deadline exceeded

Symptom: context deadline exceeded

Cause: The pod did not become Ready within the --timeout period. This is usually a pod startup issue, not a Helm issue. The release is left in the failed state.

Remedy:

  1. Check pod events and logs, as described in the other sections of this topic.
  2. Fix the underlying startup issue.
  3. Run helm upgrade again.