About Splunk sidecars

Sidecars are processes that run alongside splunkd to fulfill specific functions. They are long-running which means that they take a significant amount of time to complete and require monitoring.

Sidecars support introducing and improving capabilities in the Splunk environment. For example, several sidecars support enhanced data management in the on-premises environment. Using sidecars allows flexible feature development, efficient distribution of workloads, and improved system performance.

Sidecars are available on Splunk Enterprise and Splunk Cloud Platform. They affect your environment as follows:

You can see multiple sidecars in the process tree as subprocesses of splunkd. Sidecars can occupy network ports.
Some operating system tools, such as endpoint security scanners running in the on-premises environment, might fire alerts due to the changes related to sidecars.

How do sidecars work?

Sidecars included in Splunk Enterprise are listed in the manifest.yaml file.

Note: Sidecar processes don't include a splunk prefix in their names.

You can distinguish the following types of sidecars:

Sidecars that deploy features.
Sidecars that support sidecars that deploy features, like IPC Broker and Spotlight.

A process that manages sidecars is called the supervisor.

The splunkd process controls sidecar processes in the following way:

It starts the supervisor.
The supervisor starts and monitors sidecars and sends metrics.
The supervisor also restarts unhealthy and terminated sidecars if they are listed in the manifest.yaml file.
If splunkd stops running, the supervisor and sidecars might continue running, but are also restarted when splunkd restarts.
For example, the supervisor and sidecars might continue running in these scenarios:
- splunkd is killed or crashes.
- Sidecars do not shut down promptly after the splunk stop command is run. This delay blocks the graceful shutdown of the supervisor. As a result, splunkd terminates its direct child, the supervisor, which may leave the sidecars running. When the supervisor restarts, it detects and restarts these sidecars.

List of sidecars

The following table presents the supervisor, available sidecars, and basic information about each process.


Sidecar name	Process name	Description	Compatibility
Supervisor	`splunk-supervisor`	Starts and monitors sidecars and sends metrics.
SCIM	`identity`	Automatically deletes users removed by an administrator from the organization's Identity provider. Uses the System for Cross-domain Identity Management (SCIM) standard.	Available on the Splunk platform.
Storage	`postgres`	Helps manage the processes necessary to deploy a postgres instance. Supports enhanced data management in Splunk Enterprise.	Available only on the Linux operating system. Available for single search heads.
Data Orchestration (DO)	`cmp-orchestrator`	Coordinates running SPL2 searches, gathering results, and managing search metadata and SPL2 module files. Supports enhanced data management in Splunk Enterprise.	Available only on the Linux operating system. Available on Splunk Enterprise.
Edge Processor Control Plane	`edge-processor-config`	Enables Splunk Enterprise users to access Edge Processor features. Supports enhanced data management in Splunk Enterprise.	Available only on the Linux operating system. Available for single search heads.
OpAmp	`opamp-svc`	Manages remote agents using Open Agent Management Protocol (OpAmp), such as Edge Processor or OpenTelemetry Collector. To learn more about this protocol, see https://opentelemetry.io/docs/specs/opamp in OpenTelemetry documentation. Supports enhanced data management in Splunk Enterprise.	Available only on Linux operating system. Available for single search head and single deployment server configurations.
Agent Management	`agent-manager`	Manages a large number of different types of Splunk agents, such as the Universal forwarder. To learn more, see About agent management.	Available on Splunk Enterprise.
IPC Broker	`ipc_broker`	Handles inter-process communication between sidecars, such as assignment and discovery of ports.	Available on the Splunk platform.
Spotlight	`spotlight-collector`	On the Splunk platform, it enables metrics. On Splunk Enterprise, it collects and saves metrics from sidecars and the supervisor. Spotlight saves this data to the spotlight_metrics.ext file, which, by default, is located in the $SPLUNK_HOME/var/log/spotlight folder. Note: If you override the $SPLUNK_HOME/var/log/spotlight folder path in the configuration, the diagnostic (diag) file and the RapidDiag app might not collect metrics. To learn more about the diagnostic file, see Generate a diagnostic file and about RapidDiag, see Using RapidDiag.	Available on the Splunk platform.
Nascent	`nascent`	Ensures that the etcd service runs with the correct configuration on each search head in the cluster.	Available on the Splunk platform. Available on search head clusters.

About the Nascent sidecar

The responsibility of the Nascent sidecar is to ensure that the etcd service runs with the correct configuration on each search head in the cluster.

As a distributed key-value store, etcd keeps configuration data and enables service discovery within a cluster. It maintains data consistency across multiple nodes using the Raft consensus algorithm.

To manage the etcd cluster, Nascent performs the following:

Launches an etcd service on each search head in the SHC and configures an etcd cluster that is accessible to each search head.
Maintains the etcd cluster by ensuring that:
- A maximum of 5 search heads act as etcd cluster nodes.
- Any remaining search heads act as etcd proxy nodes.

Determining etcd cluster and etcd proxy nodes with flavors

To determine which search heads act as etcd cluster nodes and which act as etcd proxy nodes, Nascent assigns one of the following flavors to each node:

full - indicates that a search head is an etcd cluster node. It contains the etcd_data directory. Nascent runs an etcd server, that stores and replicates data, on that node.
proxy - indicates that a search head is an etcd proxy node. Nascent runs the etcd gateway, that forwards requests to etcd servers, on that node.

Nascent saves the information about assigned flavors in the node_flavor.json file.

Example assignment of flavors to nodes:

[
 {
  "name": "etcd",
  "data": {
   "full': [
    "sh1",
    "sh6",
    "sh3",
    "sh2",
    "sh4"
  ],
  "proxy": [
    "sh5",
    "sh7"
   ]
  }
 }
]