Configure TLS certificates for inter-Splunk communication

You can use transport layer security (TLS) certificates to secure connections between various Splunk components. This is known as inter-Splunk communication. Typically, Splunk processes communicate on the TCP/IP network port 8089. This port is also known as the Splunk management port.

The certificates you use can replace the default certificates that Splunk provides. You can either obtain certificates from a certificate authority, or create and sign them yourself.

Prerequisites for configuring TLS certificates for inter-Splunk communication

Before you configure TLS certificates for inter-Splunk communication, you must have the following:

  • One or more TLS certificates. You can either obtain third party certificates from a certificate authority, or create and sign them yourself. After you get the certificates, you must prepare the certificates for use with Splunk platform instances
  • Certificates in Privacy-Enhanced Mail (PEM) format that comply with the x.509 public key certificate standard
  • A private key for each certificate. Key files must comply with the Rivest-Shamir-Adelman (RSA) encryption algorithm. Splunk accepts certificates in a single PEM-format file that combines the certificate and the private key. The serverCert setting in your configuration file points to this combined file.

  • For certificates that you use with App Key Value Store (KV Store): certificates must contain x.509 version 3 (X.509v3) Extended Key Usage (EKU) certificate extensions with both the TLS Web Client (clientAuth) and TLS Web Server Authentication (serverAuth) values present. If both of these values are not present, the certificates cannot be used with KV Store. For more information about this specific requirement for KV Store, see Preparing custom certificates for use with KV store.

  • Administrative access to the Splunk Enterprise instances that you want to secure
  • An understanding of Splunk configuration files

Configure TLS certificates for communications on the Splunk management port

You can configure TLS certificates on Splunk services that use the management network port.

When you configure Splunk Enterprise to use TLS certificates, upon restart, it changes the file permissions on the certificates so that only the user that Splunk Enterprise runs as has full access. This is by design, in line with security industry standards, and cannot be changed.

  1. Open a shell or command prompt.
  2. Using this prompt or file system management tools, copy the server certificate and the certificate authority public certificate into an accessible directory on the instance where you want to configure certificates. For example, you can move the files to a destination directory of $SPLUNK_HOME/etc/auth/mycerts/.
  3. Use a text editor to open the $SPLUNK_HOME/etc/system/local/server.conf configuration file for editing.
    1. In the server.conf file, configure the instance to use the server certificate. Add the following stanzas and settings to the file.
      Setting or stanza Data type Description
      [sslConfig] stanza Sets up TLS configurations for inter-Splunk communication.
      caTrustStore (Optional) comma-separated list

      The certificate authority (CA) trust store to use when validating a network connection over TLS.

      See Configure the caTrustStore setting to determine how the Splunk platform looks for CA certificates for more information.
      caTrustStorePath (Optional; not applicable on Windows; required on Linux if caTrustStore has a value of "os") string

      The CA trust store path that the operating system uses, and that the Splunk platform is to use to find certificates for validating TLS connections.

      See Configure the caTrustStorePath setting to determine where the Splunk platform looks for CA certificates for more information.
      cipherSuite (Optional) string The cipher suite strings that TLS 1.2 sessions are to use. This setting does not apply to TLS 1.3 connections. To configure TLS 1.3 cipher suites, use the [tls1.3] stanza.
      requireClientCert (Optional) Boolean

      Whether or not the Splunk platform instance requires that a connecting client present a valid TLS certificate before the connection can succeed.

      A value of "true" means that the receiving instance must see a valid certificate to let the client authenticate. A value of "false" means that clients can connect without presenting a certificate.

      Configure this setting to "true" if you want your receivers to require authentication with certificates. When both the forwarder and receiver have a "true" value for this setting, mutually authenticated TLS (mTLS) is active. For more information on how to configure mutual TLS, see Configure mutually authenticated transport layer security (mTLS) on the Splunk platform.
      serverCert string

      The location of the server certificate on the Splunk platform instance. This is the certificate that the machine uses to support inbound connections over TLS.

      You can specify either the absolute path to the certificate, such as /opt/splunk/etc/auth/mycerts/myServerCert.pem, or you can use a relative path, such as etc/auth/mycerts/myServerCert.pem and the instance uses the Splunk platform instance installation directory.
      sslPassword (Optional) string The password that you entered when you created the private key for the server certificate, if you created a password.
      Note: Splunk Enterprise encrypts the value of the sslPassword setting in the server.conf configuration file after the first time it reads the file. The encrypted value replaces the plaintext password you set. If you did not create a password when you generated the server certificate private key, omit this setting.
      sslVersions (Optional) comma-separated list The TLS protocol versions that the instance supports for inter-Splunk communication. Supported values are "tls1.2" and "tls1.3". The default at version 10.4.0 and higher is tls1.2,tls1.3. SSLv2 and SSLv3 are always turned off. TLS 1.0 and TLS 1.1 are turned off by default in version 10.4.0 and higher.
      sslCommonNameToCheck (Optional) comma-separated list

      A list of one or more common names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the requireClientCert setting with a value of "true".

      If none of the common names in this setting value matches the common name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.
      sslAltNameToCheck (Optional) comma-separated list

      A list of one or more subject alternative names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver.

      This setting is only valid if you have configured the requireClientCert setting with a value of "true". If none of the alternate names in this setting value matches the alternate name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.
      sslRootCAPath string

      The location of the Splunk CA certificate trust store.

      The Splunk platform uses this path to find certificates to validate TLS connections depending on the value of the caTrustStore setting.

      The Splunk platform uses this setting if you give the caTrustStore setting its default value of "splunk" or "splunk,os". It does not use this setting if you give the caTrustStore setting a value of only "os". If you specify both trust stores, and an identical certificate appears in each, the Splunk platform uses the certificate in this trust store to validate the TLS connection.
      sslRootCAPathHonoredOnWindows (Deprecated) Boolean

      Whether or not the Splunk platform, when it runs on a Windows machine, uses the value in the sslRootCAPath setting when it looks for certificate authority certificates to use for TLS authentication.

      Typically, machines that run Windows use the Registry for the CA certificate trust store. You can use this setting along with the caTrustStore setting to have the instance reference the Splunk trust store instead.

      See Configure the caTrustStore setting to determine how the Splunk platform looks for CA certificates for additional information.
      [tls1.3] stanza Sets up TLS 1.3-specific configuration. Available starting in version 10.4.0. This stanza is active only when sslVersions includes "tls1.3". It has global scope — all Splunk platform clients and servers on the instance share this single stanza.
      cipherSuite (Optional; in [tls1.3]) string The TLS 1.3 cipher suites that the Splunk platform accepts for TLS 1.3 connections. Separate multiple cipher suites with a colon (:). TLS 1.3 uses a different cipher suite format than TLS 1.2 and accepts only TLS 1.3 cipher suite names. Default value: TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256.
      groups (Optional; in [tls1.3]) comma-separated list The key exchange groups that the Splunk platform uses for TLS 1.3 connections. Per RFC 8446, allowed values are: X25519, X448, secp256r1 (also known as prime256v1), secp384r1, secp521r1. Separate multiple values with a comma. Default value: prime256v1, secp384r1, secp521r1.
  4. Save the server.conf file and close it.
  5. Using the CLI, restart the splunkd process:
    CODE 
    $SPLUNK_HOME/bin/splunk restart splunkd

Configuration file examples for configuring TLS certificates on the Splunk management port

Following is an example of a server.conf configuration file on a search head and search peer. The configuration is as follows:

  • The search head uses a certificate that is in the /opt/splunk/etc/auth/mycerts directory called mySHServerCert.pem
  • The instance uses a certificate authority certificate located at /opt/splunk/etc/auth/mycerts/myCACertificate.pem
  • The server certificate was created with a password mySHCertificatePassword
  • The instance requires a certificate from clients that connect to it to ensure mutually authenticated TLS (mTLS)
  • The instance checks incoming certificates to ensure that the Common Name field in the certificate contains either splunk1.mycompany.com or splunk2.mycompany.com
CODE 
[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/mySHServerCert.pem
sslPassword = mySHCertificatePassword
requireClientCert = true
sslVersions = *,-ssl2
sslCommonNameToCheck = splunk1.mycompany.com,splunk2.mycompany.com
Note: If you supply a password for your server certificate in the server.conf file by providing a value for the sslPassword setting, the Splunk platform encrypts that password from clear text when you restart the Splunk platform instance.

Example server.conf for inter-Splunk communication with TLS 1.2 and TLS 1.3

The following example configures a search head to accept both TLS 1.2 and TLS 1.3, require a client certificate from connecting clients (mTLS), and check incoming certificates for the common name. The [tls1.3] stanza configures the TLS 1.3 cipher suites and key exchange groups. This configuration applies to Splunk Enterprise version 10.4.0 and higher.

CODE 
[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/mySHServerCert.pem
sslPassword = mySHCertificatePassword
requireClientCert = true
sslVersions = tls1.2,tls1.3
sslCommonNameToCheck = splunk1.mycompany.com,splunk2.mycompany.com

[tls1.3]
cipherSuite = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
groups = prime256v1, secp384r1, secp521r1

Manage certificate authority (CA) certificate trust stores

On Splunk Enterprise, you can determine which trust stores that the Splunk platform uses for storage of certificate authority (CA) certificates.

The Splunk platform has access to two separate trust stores:

  • The Splunk trust store. The Splunk platform uses this trust store for CA certificates to secure connections between Splunk services by default. The sslRootCAPath setting controls where this trust store is on the Splunk platform instance.
  • The operating system trust store. Every operating system has a location where you can store CA certificates. On machines that run Linux, the location depends on the distribution and version number. On machines that run Windows, the Windows Registry holds the CA certificate trust store.

To configure where the Splunk platform looks for CA certificates to use for network connections using TLS, use the caTrustStore and caTrustStorePath settings in the server.conf configuration file. The Splunk platform looks in either the Splunk trust store or the operating system trust store for your CA certificates, or both, if you specify both values for the caTrustStore setting.

If you specify the "os" value for the caTrustStore setting, then you must also specify a value for the caTrustStorePath setting. This configures the Splunk platform to locate the directory on the operating system where the OS stores its CA certificates.

Configure the caTrustStore setting to determine how the Splunk platform looks for CA certificates

The caTrustStore setting determines how the Splunk platform looks for CA certificates. There is support for the setting on machines that run Linux and Windows. It has two valid values:
Value What it means
splunk The Splunk platform looks at the file that the sslRootCAPath setting defines for the CA certificates.
os The Splunk platform looks at the file that the caTrustStorePath setting defines for the CA certificates on machines that run Linux and in the Registry on machines that run Windows.
splunk,os The Splunk platform checks both the Splunk trust store and the operating system trust store for CA certificates. If an identical certificate appears in both stores, the Splunk platform uses the certificate in the Splunk trust store.
You can specify either or both values for the setting. What you specify and the operating system that you run determines the logic that the Splunk platform uses to locate the certificates. The following table shows the certificate location logic.
caTrustStore value Operating System Where the Splunk platform looks for CA certificates
splunk Linux The location defined by the sslRootCAPath setting
Windows If the sslRootCAPathHonoredOnWindows setting has a value of "true": The location defined by the sslRootCAPath setting
Otherwise: It does not look for certificates
splunk,OS Linux The locations defined by the sslRootCAPath setting and the caTrustStorePath setting
Windows The Windows Registry and, if the sslRootCAPathHonoredOnWindows setting has a value of "true", the location defined by the sslRootCAPath setting
OS Linux The location defined by the caTrustStorePath setting
Windows The Windows Registry
If you specify splunk,OS as values, and the Splunk platform finds an identical TLS certificate in both trust stores, it uses the certificate in the Splunk trust store to validate the TLS connection.

Configure the caTrustStorePath setting to determine where the Splunk platform looks for CA certificates

Different Linux distributions use different locations for the CA trust store. The caTrustStorePath setting determines how the Splunk platform looks for the trust store, based on the distribution of Linux you run.

There is support for the setting on machines that run Linux only. It is not applicable for Windows, as Windows stores its certificates in the Windows Registry in all cases except when the sslRootCAPathHonoredOnWindows setting has a "true" value.

If caTrustStore has a value of "OS", but this setting has either no value or an invalid value, then the Splunk platform does not attempt to load any certificates from the OS trust store to validate TLS, and logs an error message in the splunkd.log log file.

There is no default value for this setting, which means you must give it one if you give the caTrustStore setting values of either "splunk,OS" or "OS".

Following are example trust store locations for popular Linux distributions:
Linux distribution Trust store location
Debian, Ubuntu, Gentoo: all versions /etc/ssl/certs/ca-certificates.crt
Fedora, Red Hat Enterprise Linux (RHEL): versions 6, 8, and 9 /etc/pki/tls/certs/ca-bundle.crt
CentOS, RHEL: version 7 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Prerequisites for certificate usage

There are limitations to the kinds of CA certificates you can reference in the configuration.

Certificates that meet the requirements include:

  • Root certificates with a basicConstraints X.509 V3 extension value of any of the following: "CA:FALSE", "CA:TRUE", or "none".
  • Self-signed CA certificates
  • Certificates that an intermediate CA issues that have a basicConstraints value of "CA:TRUE" as long as all intermediate CA certificates in the certificate chain also have a value of "CA:TRUE"
  • Any certificate chain where all of the certificates in the chain meet all of the previously described requirements

Certificates that do not meet the requirements include:

  • Any root certificate issued by a root CA and whose basicConstraints value has a value of "CA:FALSE"
  • Any certificate or certificate chain that an intermediate CA self-signs