Sidecar troubleshooting

Monitor the status of sidecars and troubleshoot any issues. Learn which log files are associated with each sidecar.

To track the status of sidecars, use the splunkd health report. To learn more about this tool, see About proactive Splunk component monitoring.

If an issue with a sidecar occurs, try one or more of the following steps:

  • Check the sidecar configuration and modify settings, if necessary.
  • Restart all sidecars.

    Even if the issue applies to an individual sidecar, you need to restart all sidecars. You cannot restart sidecars individually.

  • If the issue still occurs, generate a diagnostic (diag) file and send it to Splunk support for further assistance in troubleshooting the issue. To learn more about a diagnostic file, see Generate a diagnostic file.

Troubleshoot with log files

The logs that the supervisor and sidecars generate are saved in the $SPLUNK_HOME/var/log/splunk directory. The following list presents the logs and data that they capture.

Note: By generating a diagnostic (diag) file, you can retrieve data from stdout log files.
Log file name Description
supervisor.log Logs from the supervisor about the supervisor itself, sidecar lifecycle and health, and endpoint registration.
sup-pkg-identity-stdout.log System logs for the SCIM sidecar.
postgres-*.log Hourly logs from the PostgreSQL database that is managed by the Storage sidecar. The log file retains up to 24 logs, or 1 per hour.

sup-pkg-postgres.log

sup-pkg-postgres-stdout.log

 System logs for the Storage sidecar.
spl2-orchestrator.log Logs of requests for the SPL2 language server that is managed by the Data Orchestration sidecar.

sup-pkg-cmp-orchestrator.log

sup-pkg-cmp-orchestrator-stdout.log

 System logs for the Data Orchestration sidecar.

sup-pkg-edge-processor-config-stdout.log

sup-pkg-opamp-svc-stdout.log

 Service logs for the Edge Processor Control Plane sidecar.
sup-pkg-agent-manager-stdout.log System logs for the Agent Management sidecar.
sup-pkg-ipc_broker-stdout.log System logs for the IPC Broker sidecar.

sup-pkg-spotlight-collector.log

sup-pkg-spotlight-collector-stdout.log

 System logs for the Spotlight sidecar.

sup-pkg-nascent.log

sup-pkg-nascent-stdout.log

 System logs for the Nascent sidecar.

etcd.log

 The log for etcd. Useful when troubleshooting the Nascent sidecar.

splunkd.log

 The primary log file for Splunk Enterprise. It contains system logs for the Nascent sidecar running on an etcd proxy node.

Sidecars - troubleshooting tips

Issues with sidecars may result from the following:

Errors in ingesting, processing and routing data

Follow these tips to troubleshoot issues:
  • Ensure that an Edge Processor instance is healthy.

  • Ensure that users can connect to the Edge Processor instance.

  • Ensure that your firewall is not causing a connection or data ingestion error.

Errors in Edge Processors, pipelines, data destinations, and source types

Follow these tips to troubleshoot issues:

  • If the Storage sidecar is not starting up, ensure that you run Splunk Enterprise as a non-root user. See Sidecars - troubleshooting tips.

    Issues with starting up the Storage process can cause issues with other sidecars.

  • Check that sidecar issues are not caused by network issues related to a firewall, proxy, or DNS.
  • If an Edge Processor instance is disconnected for more than 3 hours, it might no longer be trusted. Provision it again to reestablish the instance as trusted.