Troubleshoot Enterprise Security installation

Troubleshoot Enterprise Security installation issues.

Use the following logs and troubleshooting guidance to diagnose ES app installation issues.

Checking installation progress

The following logs can be useful to check the ES app installation progress.

Splunk operator log:

CODE 
kubectl logs <operator_pod_name>

Logs of the respective pods:

CODE 
kubectl logs <pod_name>

Check the pod log (for example, the deployer pod) if you want to monitor the pod while it comes up to a ready state or if it has gone to an error state.

ES post-install failures

Check the operator log for one or more entries with the following content: premium scoped app package install failed followed by specific failure reasons.

Error when using strict mode for sslEnablement and Splunk Web is not SSL enabled:

JSON 
2022-12-07T00:17:36.780549729Z  ERROR   handleEsappPostinstall  premium scoped app package install failed   {"controller": "searchheadcluster", "controllerGroup": "enterprise.splunk.com", "controllerKind": "SearchHeadCluster", "SearchHeadCluster": {"name":"shc1","namespace":"default"}, "namespace": "default", "name": "shc1", "reconcileID": "83133a29-ca0d-46cc-9ae5-6f26385d4506", "name": "shc1", "namespace": "default", "pod": "splunk-shc1-deployer-0", "app name": "splunk-enterprise-security_702.spl", "stdout": "", "stderr": "FATAL: Error in 'essinstall' command: You must have SSL enabled to continue\n", "post install command": "/opt/splunk/bin/splunk search '| essinstall --ssl_enablement strict --deployment_type shc_deployer' -auth admin:`cat /mnt/splunk-secrets/password`", "failCount": 1, "error": "command terminated with exit code 17"}

To fix this error, choose one of the following options:

  1. Enable Splunk Web SSL. See the section on enabling strict mode for sslEnablement in SHC in the installation topic.
  2. Use sslEnablement=ignore.

Connection timeout error:

JSON 
2022-12-07T00:48:11.542927588Z	ERROR	handleEsappPostinstall	premium scoped app package install failed	{"controller": "searchheadcluster", "controllerGroup": "enterprise.splunk.com", "controllerKind": "SearchHeadCluster", "SearchHeadCluster": {"name":"shc1it","namespace":"default"}, "namespace": "default", "name": "shc1it", "reconcileID": "b34966a2-e716-428f-a0c6-7611812e6b24", "name": "shc1it", "namespace": "default", "pod": "splunk-shc1it-deployer-0", "app name": "splunk-enterprise-security_702.spl", "stdout": "", "stderr": "FATAL: Error in 'essinstall' command: (InstallException) \"install_apps\" stage failed - Splunkd daemon is not responding: ('Error connecting to /services/apps/shc/es_deployer: The read operation timed out',)\n", "post install command": "/opt/splunk/bin/splunk search '| essinstall --ssl_enablement ignore --deployment_type shc_deployer' -auth admin:`cat /mnt/splunk-secrets/password`", "failCount": 2, "error": "command terminated with exit code 17"}

To fix this error, check the following areas:

  1. Check the splunkd log in the deployer pod for any issues.
  2. Check the splunkdConnectionTimeout setting in web.conf.

Common issues

  • ES installation failed because you used the default sslEnablement mode (strict). Enable Splunk Web SSL in web.conf. See the section on enabling strict mode for sslEnablement in SHC in the installation topic.
  • Ansible task timeouts. Raise associated timeout values (splunkdConnectionTimeout in web.conf, rcv_timeout, send_timeout, cxn_timeout values in server.conf).
  • Pod recycles. Raise the livenessProbe value.