Configure mutually authenticated transport layer security (mTLS) on the Splunk platform

You can configure mutually authenticated transport layer security (otherwise known as mTLS) on certain networked connections on the Splunk platform. mTLS provides an increased level of security between Splunk platform nodes by ensuring, among other things, that both sides of the network connection share the same private encryption key. TLS certificates on both sides provide additional verification and further establish that both ends of the connection are who they claim to be. This significantly reduces the attack surface for common vulnerabilities such as machine-in-the-middle (MITM) or ransomware attacks.

Splunk component network connections that you can secure with mTLS

On Splunk Enterprise, you can secure the following types of network connections with mTLS:

Client node Server node Method Network port
HTTP Event Collector (HEC) Indexer - HEC HTTP over Transmission Control Protocol (TCP) 8088
Forwarder Indexer Splunk-to-Splunk (S2S) over TCP 9997
Web user (through Common Identification Card / Personal Identity Verification (CAC/PIV) Search head HTTP over TCP 443
Search head cluster member replication Search head cluster member replication TCP 8081, 8181, 9887
Indexer cluster peer node replication Indexer cluster peer Node replication TCP 8080, 9887
Indexer cluster manager Indexer cluster peer node HTTP over TCP 8089
Enterprise Security / Behavior Analytics (ESBA) client Splunk Cloud Services (SCS) server S2S over TCP N/A
App Key Value Store (KV Store) Search head cluster member HTTP over TCP 8089
Indexer (REST API) Search head (REST API) HTTP over TCP 8089
Federated Search (REST API) Splunk management port (REST API) HTTP over TCP 8089

After you configure both sides of the network connection with mTLS, the connections are secured for as long as the TLS certificates that each member presents are valid. When TLS certificates expire, you can renew them.

Note: The Splunk platform bypasses mTLS on connections that involve only the local machine (localhost), such as when you use the simpleRequest REST API function. Starting in Splunk Enterprise version 10.6, the Splunk command line interface (CLI) supports client certificate authentication when connecting to remote instances. See Use the Splunk CLI with client certificate authentication.

Prerequisites for configuring mTLS on Splunk platform nodes

Before you try to configure your Splunk platform nodes for mutual TLS, you must have the following items in place in your Splunk platform deployment:

  • One or more TLS certificates for the client and the server side of the connection.
    • You can either obtain third-party certificates, or generate them yourself
    • You can use the same certificates for each side of the connection
    • After you get the certificates, you must prepare the certificates for use with Splunk platform instances
    • The certificate files must be in Privacy-Enhanced Mail (PEM) format and comply with the X.509 public key certificate standard
    • You must have a private key file for each certificate file. The certificates must share the same private key.
    • The key files that come with the certificates must be in RSA security format.
  • At least one client and one server connection where you want to turn on mTLS
    • There is support only for the Splunk components that appear in the table in the previous section

Configure mTLS for Splunk platform nodes

The following procedure is a general one for turning on mTLS. The specific tasks that are necessary to turn on mTLS depend on your specific configuration.

The requireClientCert setting is the main setting that turns on mTLS for supported network connections. The setting determines whether the Splunk platform instance requires that a connecting client present a valid TLS certificate before the connection can succeed.

A value of "true" for the setting means that the receiving instance must see a valid certificate to let the client authenticate. A value of "false" means that clients can connect without presenting a certificate.

Configure this setting to "true" on the receiver (the node that is receiving the connection) if you want your receivers to require authentication with certificates.

Note: The [sslConfig] stanza in server.conf is a global stanza that applies to all connections on the instance. Individual communication channels, such as Splunk-to-Splunk (S2S) forwarding, also have their own requireClientCert setting in their respective stanzas, such as the tcp-ssl stanza. Per-channel settings take precedence over the global [sslConfig] setting for that channel.
  1. Confirm that you have properly formatted your certificates for use on the Splunk platform and concatenated them. See How to prepare TLS certificates for use with the Splunk platform.
  2. Install the certificates into the correct directory in your Splunk platform installation on both nodes whose connection you want to secure with mTLS.
    Note: You can also install certificates in the operating system local certificate store.
  3. On the receiving node, configure the Splunk platform to use the new certificates and turn on mTLS.
    1. Open the $SPLUNK_HOME/etc/system/local/server.conf file for editing.
    2. In the [sslConfig] stanza for the file, add the following line to the configuration:
      CODE 
      requireClientCert = true
    3. (Optional) If you have not already, reference the TLS certificate that you installed using the sslRootCAPath setting:
      CODE 
      sslRootCAPath = <location of certificates in PEM format>
    4. Save the file and close it.
    5. Restart the Splunk platform instance.

Use the Splunk CLI with client certificate authentication

Supply a dedicated client certificate and private key when you use the Splunk CLI to connect an instance to a remote management port that requires client certificate authentication.

Starting in Splunk Enterprise version 10.6, when you connect an instance to a remote management port that requires client certificate authentication, you can supply the client certificate and private key explicitly using Splunk command line interface (CLI) options.

Previously, the CLI reused the web certificate as its client certificate. DigiCert, a commercial certificate authority, no longer includes the client authentication extended key usage (EKU) in its certificates, which prevents the CLI from reusing the web certificate. You must now supply a dedicated client certificate using the arguments described in this topic when you connect an instance to a management port that requires client certificate authentication.

You must use the following CLI arguments for client certificate authentication when you use the -uri argument to connect to a remote Splunk instance. You do not need them for local commands.

Note: This change applies to the CLI only. Apps that use the Splunk SDK or the simpleRequest REST API function still reuse the web certificate (web.pem) as the client certificate when connecting to a remote management port. Splunk Enterprise version 10.6 does not support supplying a dedicated client certificate for those app connections.

CLI arguments for client certificate authentication

The following table describes the arguments you use to supply client certificate information to the CLI:

Argument Required Description
--client-cert path Yes, to use mTLS The path to the Privacy-Enhanced Mail (PEM) file that contains the client certificate. If you omit --client-key, the CLI assumes the private key resides in this file.
--client-key path No The path to the PEM file that contains the private key for the client certificate. If you omit this argument, the CLI looks for the private key in the file you specify in --client-cert.
--client-key-password-file path No The path to a file whose first line contains the plaintext passphrase for an encrypted private key. If you omit this argument, the CLI falls back to the SPLUNK_CLIENT_KEY_PASSWORD environment variable. If you set neither, the CLI assumes the private key has no encryption.

Example:

CODE 
splunk <command> -uri https://<host>:8089 \
  --client-cert /etc/splunk/certs/client.pem \
  --client-key /etc/splunk/certs/client_key.pem
Note: The remote Splunk instance must trust the certificate authority (CA) that issued your client certificate. Add the issuing CA to the CA file referenced by the sslRootCAPath setting in the server.conf configuration file on the remote instance.

Specify paths to certificate files

Paths for these arguments support environment variable expansion. For example, $CERT_DIR/client.pem is valid. Paths can be relative or absolute. Relative paths resolve against the current working directory.

Examples:

PYTHON 
# Running from the directory where the certs live:
cd /etc/splunk/certs
splunk <command> -uri https://<host>:8089 \
  --client-cert client.pem \
  --client-key client_key.pem

# Running from outside the cert directory:
splunk <command> -uri https://<host>:8089 \
  --client-cert /etc/splunk/certs/client.pem \
  --client-key /etc/splunk/certs/client_key.pem

Password resolution for encrypted private keys

When you supply an encrypted private key, the CLI determines the passphrase to use in the following order:

  1. --client-key-password-file. The CLI reads the first line of the file as the passphrase. The CLI treats an empty file as no passphrase and suppresses the SPLUNK_CLIENT_KEY_PASSWORD environment variable for child processes.
  2. SPLUNK_CLIENT_KEY_PASSWORD environment variable. The CLI uses this as a fallback when you do not provide a password file.
  3. Neither set. The CLI assumes the private key has no encryption.
Note: Splunk Enterprise limits passwords to 63 bytes.