Sidecar troubleshooting
To track the status of sidecars, use the splunkd health report. To learn more about this tool, see About proactive Splunk component monitoring.
If an issue with a sidecar occurs, try one or more of the following steps:
- Check the sidecar configuration and modify settings, if necessary.
- Restart all sidecars.
Even if the issue applies to an individual sidecar, you need to restart all sidecars. You cannot restart sidecars individually.
- If the issue still occurs, generate a diagnostic (diag) file and send it to Splunk support for further assistance in troubleshooting the issue. To learn more about a diagnostic file, see Generate a diagnostic file.
Troubleshoot with log files
The logs that the supervisor and sidecars generate are saved in the $SPLUNK_HOME/var/log/splunk directory. The following list presents the logs and data that they capture.
| Log file name | Description |
|---|---|
| supervisor.log | Logs from the supervisor about the supervisor itself, sidecar lifecycle and health, and endpoint registration. |
| sup-pkg-identity-stdout.log | System logs for the SCIM sidecar. |
| postgres-*.log | Hourly logs from the PostgreSQL database that is managed by the Storage sidecar. The log file retains up to 24 logs, or 1 per hour. |
sup-pkg-postgres.log sup-pkg-postgres-stdout.log | System logs for the Storage sidecar. |
| spl2-orchestrator.log | Logs of requests for the SPL2 language server that is managed by the Data Orchestration sidecar. |
sup-pkg-cmp-orchestrator.log sup-pkg-cmp-orchestrator-stdout.log | System logs for the Data Orchestration sidecar. |
sup-pkg-edge-processor-config-stdout.log sup-pkg-opamp-svc-stdout.log | Service logs for the Edge Processor Control Plane sidecar. |
| sup-pkg-agent-manager-stdout.log | System logs for the Agent Management sidecar. |
| sup-pkg-ipc_broker-stdout.log | System logs for the IPC Broker sidecar. |
sup-pkg-spotlight-collector.log sup-pkg-spotlight-collector-stdout.log | System logs for the Spotlight sidecar. |
Sidecars - troubleshooting tips
Issues with sidecars may result from the following:
Errors in ingesting, processing and routing data
- Ensure that an Edge Processor instance is healthy.
Ensure that users can connect to the Edge Processor instance.
-
Ensure that your firewall is not causing a connection or data ingestion error.
Errors in Edge Processors, pipelines, data destinations, and source types
If the Storage sidecar is not starting up, ensure that you run Splunk Enterprise as a non-root user. See Run Splunk Enterprise as a different or non-root user.
Issues with starting up the Storage process can cause issues with other sidecars.
- Check that sidecar issues are not caused by network issues related to a firewall, proxy, or DNS.
- If an Edge Processor instance is disconnected for more than 3 hours, it might no longer be trusted. Provision it again to reestablish the instance as trusted.