Run Splunk Enterprise as a systemd service
Splunk Enterprise provides support for Linux systemd with an enhanced enable boot-start command that lets you automatically configure systemd to manage splunkd as a service.
What is systemd?
systemd is a system startup and service manager that is widely deployed as the default init system on most major Linux distributions. You can configure systemd to manage processes, such as splunkd, as services, and allocate system resources to those processes under cgroups.
systemd advantages
systemd offers the following general advantages:
- Enhanced parallel processing.
- Simplified configuration with standardized unit text files. No scripts required.
- Improved mechanism for expressing dependencies. For example, you can specify in the unit file that the network must be up before startup of the
splunkdservice occurs.
systemd offers these additional specific advantages for Splunk deployments:
- Start
splunkdat boot. - Monitor and manage
splunkdservice during runtime. - Provides tools to debug and troubleshoot boot-time and service activities.
- Allows more control over plug-in monitoring tools that track the status of Splunk instances.
- Simplifies the set up of
cgroupsrequired for workload management in Splunk Enterprise. See Set up Linux for workload management in the Workload Management manual.
Configure systemd to manage splunkd
You can use either of the following two methods to configure systemd to manage splunkd as a service:
systemd using enable boot-start, a Splunk service unit file is created automatically. No additional manual configuration is required.System requirements
For a list of supported Linux distributions, see System requirements for use of Splunk Enterprise on-premises.
- To configure
systemdusingenable boot-startrequires Splunk Enterprise version 7.2.2 or later. - To enable workload management in Splunk Enterprise under
systemdrequiressystemdversion 219 or higher. For more information, see Linux operating system requirements in the Workload Management manual. - Splunk Enterprise 9.4 and higher supports Linux cgroups version 2 by default.
Permissions requirements
The enable boot-start command and systemd have the following permissions requirements:
- Non-root users must have super user permissions to configure
systemdusingenable boot-start. - Non-root users must have super user permissions to run
splunk start|stop|restartoperations undersystemd.
Unprivileged users must use sudo to run splunk start|stop|restart. If you do not use sudo and attempt to run splunk start|stop|restart when managed by systemd, a prompt appears requesting authentication. For example:
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Multiple identities can be used for authentication:
1. <username_1>
2. <username_2>
Choose identity to authenticate as (1-2): 2
Password:
==== AUTHENTICATION COMPLETE ===
Alternately, you can install polkit rules with the enable boot-start command to allow unprivileged users to run start|stop|restart operations under systemd without using sudo. For instructions, see Install polkit rules to elevate user permissions.
Unit file naming considerations
The enable boot-start command creates a systemd unit file named Splunkd.service. The unit file name is based on the SPLUNK_SERVER_NAME in splunk-launch.conf, which is set by default to Splunkd.
If for any reason you remove the SPLUNK_SERVER_NAME value from splunk-launch.conf, enable boot-start creates a unit file named splunkd.service (lower case "splunkd") and sets SPLUNK_SERVER_NAME=splunkd in the splunk-launch.conf file.
You can specify a different name of your choice for the unit file when you run enable boot-start. See Specify a different unit file name.
Configure systemd using enable boot-start
You can configure systemd to manage splunkd as a service using the enable boot-start command, as follows:
- Log into the machine on which you want to configure
systemdto managesplunkdas a service. - Stop
splunkd.$SPLUNK_HOME/bin/splunk stop - If you previously enabled Splunk Enterprise to start at boot using the
enable boot-startcommand, rundisable boot-startto remove thesplunkinitscript located in/etc/init.dand its symbolic links.[sudo] $SPLUNK_HOME/bin/splunk disable boot-startFor instructions on how to reinstall the
splunkinitscript, see Install splunk init script. - Run the
enable boot-startcommand, specifying the-systemd-managed,-user, and-groupparameters, as follows:[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname>Note: Specifying-userand-groupis optional but recommended. If you do not specify-user, theSPLUNK_OS_USERinsplunk-launch.confis used. IfSPLUNK_OS_USERis not defined, the owner of thesplunkbinary is used.This installs the
systemdservice unit file for cgroups v2, namedSplunkd.serviceby default, in/etc/systemd/system. For example:#This unit file replaces the traditional start-up script for systemd #configurations, and is used when enabling boot-start for Splunk on #systemd-based Linux distributions. [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network-online.target Wants=network-online.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd KillMode=mixed KillSignal=SIGINT TimeoutStopSec=360 LimitNOFILE=65536 LimitRTPRIO=99 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=splunk Group=splunk Delegate=true CPUWeight=100 MemoryMax=8036175872 PermissionsStartOnly=true ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n" [Install] WantedBy=multi-user.targetNote: By default, theMemoryMaxvalue is set to the total system memory available in bytes when the service unit file is created. TheMemoryMaxvalue will not update if the total available system memory changes. To update theMemoryMaxvalue in the unit file, you must manually edit the value or use theboot-startcommand to disable and re-enable systemd.CAUTION: The following unit file properties are required. Do not change these values without appropriate guidance.Type=simpleRestart=alwaysExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd-
Delegate=trueThis property is required for workload management. See Configure workload management
CAUTION: Do not use the following properties. These properties can causesplunkdto fail on restart.-
RemainAfterExit=yes -
ExecStop
For more information, see Systemd unit file properties.
- Start
splunkd.[sudo] $SPLUNK_HOME/bin/splunk startThis starts
splunkdas asystemdservice.Note: Undersystemd,splunk start|stop|restartcommands are mapped tosystemctl start|stop|restartcommands. - Verify that
splunkdis running as asystemdservice. For example:$SPLUNK_HOME/bin/splunk status splunkd is running (PID: 24772). splunk helpers are running (PIDs: 24843 24857 24984 25032).Note: Alternatively, you can usesystemctl statusto check if thesplunkdprocess is running. However, when using this command, a brief time lag can occur during whichsystemctl statusshows "active" andsplunk statusshows "splunkd is not running". - For distributed deployments, repeat steps 1-6 on all search heads and indexers.
Additional options for enable boot-start
The enable boot-start command supports these additional options:
Install splunk init script
In version 7.2.2 and higher, the enable boot-start command adds a -systemd-managed 0|1 option that controls whether to install the splunk init script in /etc/init.d or the Splunkd.service unit file in /etc/systemd/system.
To install the splunk init script, specify -systemd-managed 0:
$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user <username>
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
See Configure Splunk Enterprise to start at boot time.
-systemd-managed option, the enable boot-start command defaults to -systemd-managed 1 and installs the Splunkd.service unit file. In version 7.3.0 and later, this default behavior is reversed, and the enable boot-start command defaults to -systemd-managed 0 and installs the splunkinit file.init.d boot-start script is not compatible with RHEL 8 and higher.Specify a different unit file name
The default splunkd unit file name is Splunkd.service. You can specify a different name for the unit file and update the SPLUNK_SERVER_NAME value in splunk-launch.conf using the -systemd-unit-file-name option. For example, to create a unit file with the name "splunk.service":
$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -systemd-unit-file-name splunk
Systemd unit file installed at /etc/systemd/system/splunk.service.
Configured as systemd managed service.
For more information, see Unit file naming considerations.
Install polkit rules to elevate user permissions
In version 8.1.1 and higher, the enable boot-start command adds an option to install polkit rules that allow non-root users to run start, stop, and restart operations under systemd without using sudo. Installing the polkit rules can reduce overhead for admins that must otherwise add unprivileged users to the sudoers file to run these operations under systemd.
To install polkit rules:
Run the enable boot-start command, specifying the -create-polkit-rules option, as follows:
./splunk enable boot-start -systemd-managed 1 -create-polkit-rules 1 -user <username>
If you previously ran enable boot-start and specified a different user, you must change the owner of $SPLUNK_HOME to the new user for whom you create the polkit rules. For example:
chown -R <username> $SPLUNK_HOME
create-polkit-rules option, you must install the Polkit library on your system if you have not already done so.Configure systemd on a clean install
To configure systemd on a clean installation of Splunk Enterprise:
-
Expand the install package in an appropriate directory. For example:
tar xvfz splunk_package_name.tgz -C /opt - Run
enable boot-startto install theSplunkd.serviceunit file:sudo $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username>Note: When running enable boot-start for the first time after a clean install, Splunk Enterprise prompts you to accept the Splunk software license agreement. To automatically accept the license without prompt, specify the--accept-licenseflag with the command. -
Start
splunkd.sudo $SPLUNK_HOME/bin/splunk start -
Verify that
splunkdis running as asystemdservice.$SPLUNK_HOME/bin/splunk status
Manage clusters under systemd
When managing an indexer cluster under systemd:
- You must use the
sudocommand to start, stop, and restart the cluster manager node or individual peer nodes usingsplunk start|stop|restartcommands. - You do not need
sudoto perform a rolling restart using thesplunk rolling-restart cluster-peerscommand, or to take a peer offline using thesplunk offlinecommand.
When managing a search head cluster under systemd:
- You must use the
sudocommand to start, stop, and restart cluster members usingsplunk start|stop|restartcommands. - You do not need
sudoto perform a rolling restart using thesplunk rolling-restart shcluster-memberscommand, or to remove a cluster member using thesplunk remove shcluster-memberscommand.