Configure TLS certificates for inter-Splunk communication

You can use transport layer security (TLS) certificates to secure connections between various Splunk components. This is known as inter-Splunk communication. Typically, Splunk processes communicate on the TCP/IP network port 8089. This port is also known as the Splunk management port.

The certificates you use can replace the default certificates that Splunk provides. You can either obtain certificates from a certificate authority, or create and sign them yourself.

Prerequisites for configuring TLS certificates for inter-Splunk communication

Before you can secure communications between Splunk indexers and forwarders, you must have the following:

  1. One or more TLS certificates.
    • You can either obtain third party certificates from a certificate authority, or create and sign them yourself
    • After you get the certificates, you must prepare the certificates for use with Splunk platform instances
    • The certificates must be in Privacy-Enhanced Mail format and comply with the x.509 public key certificate standard
    • You must have a private key file for each certificate file.
    • The key files that come with the certificates must comply with the Rivest-Shamir-Adelman (RSA) security algorithm.

    • For certificates that you use with App Key Value Store (KV Store), the certificates must contain x.509 version 3 (X.509v3) Extended Key Usage (EKU) certificate extensions with both the TLS Web Client (clientAuth) and TLS Web Server Authentication (serverAuth) values present. If both of these values are not present, the certificates cannot be used with KV Store. For more information about this specific requirement for KV Store, see Preparing custom certificates for use with KV store.

  2. Administrative access to the Splunk Enterprise instances that you want to secure
  3. An understanding of Splunk configuration files

Configure TLS certificates for communications on the Splunk management port

You can configure TLS certificates on Splunk services that use the management network port.

When you configure Splunk Enterprise to use TLS certificates, upon restart, it changes the file permissions on the certificates so that only the user that Splunk Enterprise runs as has full access. This is by design, in line with security industry standards, and cannot be changed.

  1. Open a shell or command prompt.
  2. Using this prompt or file system management tools, copy the server certificate and the certificate authority public certificate into an accessible directory on the instance where you want to configure certificates. For example, you can move the files to a destination directory of $SPLUNK_HOME/etc/auth/mycerts/.
  3. Use a text editor to open the $SPLUNK_HOME/etc/system/local/server.conf configuration file for editing.
    1. In the server.conf file, configure the instance to use the server certificate. Add the following stanzas and settings to the file.
      Setting/stanza name Data type Description
      [sslConfig] n/a Sets up TLS/SSL configurations for inter-Splunk communication.
      caTrustStore (Optional) comma-separated list

      The certificate authority (CA) trust store to use when validating a network connection over TLS.

      See Configure the caTrustStore setting to determine how the Splunk platform looks for CA certificates for more information.

      caTrustStorePath (Optional, not applicable on machines that run Windows, required on machines that run Linux if 'caTrustStore' has a value of "os") string

      The certificate authority (CA) trust store path that the operating system uses, and that the Splunk platform is to use to find certificates for validating TLS connections.

      See Configure the caTrustStorePath setting to determine where the Splunk platform looks for CA certificates for more information.

      cipherSuite (Optional) string The list of cipher suite strings that the TLS/SSL sessions are to use.
      requireClientCert (Optional) Boolean

      Whether or not the Splunk platform instance requires that a connecting client present a valid TLS certificate before the connection can succeed.

      A value of "true" means that the receiving instance must see a valid certificate to let the client authenticate. A value of "false" means that clients can connect without presenting a certificate.

      Configure this setting to "true" if you want your receivers to require authentication with certificates. When both the forwarder and receiver have a "true" value for this setting, mutually authenticated TLS or mTLS is active. For more information on how to configure mutual TLS, see Configure mutually authenticated transport layer security (mTLS) on the Splunk platform.

      serverCert string

      The location of the server certificate on the Splunk platform instance. This is the certificate that the machine uses to support inbound connections over TLS/SSL.

      You can specify either the absolute path to the certificate, such as /opt/splunk/etc/auth/mycerts/myServerCert.pem, or you can use a relative path, such as etc/auth/mycerts/myServerCert.pem and the instance uses the Splunk platform instance installation directory.

      sslPassword (Optional) string The password that you entered when you created the certificate, if you created a password.
      sslVersions (Optional) comma-separated list The list of TLS/SSL versions that the receiver supports. The Splunk platform supports the following versions for SSL and TLS: "ssl3", "tls1.0", "tls1.1", and "tls1.2".
      sslCommonNameToCheck (Optional except in certain circumstances) comma-separated list

      A list of one or more common names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver. This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true".

      If none of the common names in this setting value matches the common name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.

      sslAltNameToCheck (Optional except in certain circumstances) comma-separated list

      A list of one or more subject alternative names upon which the receiving Splunk platform instance checks for a match in the certificate that the client presents upon connecting to the receiver.

      This setting is only valid if you have configured the 'requireClientCert' setting with a value of "true". If none of the alternate names in this setting value matches the alternate name in the certificate of the connecting client, the receiving instance declines the connection as not authorized.

      sslRootCAPath string

      The location of the Splunk CA certificate trust store.

      The Splunk platform uses this path to find certificates to validate TLS connections depending on the value of the 'caTrustStore' setting.

      The Splunk platform uses this setting if you give the 'caTrustStore' setting its default value of "splunk" or "splunk,os". It does not use this setting if you give the 'caTrustStore' setting a value of only "os". If you specify both trust stores, and an identical certificate appears in each, the Splunk platform uses the certificate in this trust store to validate the TLS connection.

      sslRootCAPathHonoredonWindows (Deprecated)Boolean

      Whether or not the Splunk platform, when it runs on a Windows machine, uses the value in the sslRootCAPath setting when it looks for certificate authority certificates to use for TLS authentication.

      Typically, machines that run Windows use the Registry for the CA certificate trust store. You can use this setting along with the caTrustStore setting to have the instance reference the Splunk trust store instead.

      See Configure the caTrustStore setting to determine how the Splunk platform looks for CA certificates for additional information.

  4. Save the server.conf file and close it.
  5. Using the CLI, restart the splunkd process: 
    # $SPLUNK_HOME/bin/splunk restart splunkd

Configuration file examples for configuring TLS certificates on the Splunk management port

Following is an example of a server.conf configuration file on a search head and search peer. The configuration is as follows:

  • The search head uses a certificate that is in the /opt/splunk/etc/auth/mycerts directory called mySHServerCert.pem
  • The instance uses a certificate authority certificate located at /opt/splunk/etc/auth/mycerts/myCACertificate.pem
  • The server certificate was created with a password "mySHCertificatePassword"
  • The instance requires a certificate from clients that connect to it to ensure mutually authenticated TLS (mTLS)
  • The instance checks incoming certificates to ensure that the Common Name field in the certificate contains either "splunk1.mycompany.com" or "splunk2.mycompany.com"
[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/mySHServerCert.pem
sslPassword = mySHCertificatePassword
requireClientCert = true
sslVersions = *,-ssl2
sslCommonNameToCheck = splunk1.mycompany.com,splunk2.mycompany.com

Note: If you supply a password for your server certificate in the server.conf file by providing a value for the sslPassword setting, the Splunk platform encrypts that password from clear text when you restart the Splunk platform instance.

Manage certificate authority (CA) certificate trust stores

On Splunk Enterprise, you can determine which trust stores that the Splunk platform uses for storage of certificate authority (CA) certificates.

The Splunk platform has access to two separate trust stores:

  • The Splunk trust store. The Splunk platform uses this trust store for CA certificates to secure connections between Splunk services by default. The sslRootCAPath setting controls where this trust store is on the Splunk platform instance.
  • The operating system trust store. Every operating system has a location where you can store CA certificates. On machines that run Linux, the location depends on the distribution and version number. On machines that run Windows, the Windows Registry holds the CA certificate trust store.

To configure where the Splunk platform looks for CA certificates to use for network connections using TLS, use the caTrustStore and caTrustStorePath settings in the server.conf configuration file. The Splunk platform looks in either the Splunk trust store or the operating system trust store for your CA certificates, or both, if you specify both values for the caTrustStore setting.

If you specify the "os" value for the caTrustStore setting, then you must also specify a value for the caTrustStorePath setting. This configures the Splunk platform to locate the directory on the operating system where the OS stores its CA certificates.

Configure the caTrustStore setting to determine how the Splunk platform looks for CA certificates

The caTrustStore setting determines how the Splunk platform looks for CA certificates. There is support for the setting on machines that run Linux and Windows. It has two valid values:
ValueWhat it means
splunkThe Splunk platform looks at the file that the sslRootCAPath setting defines for the CA certificates.
OSThe Splunk platform looks at the file that the caTrustStorePath setting defines for the CA certificates on machines that run Linux and in the Registry on machines that run Windows.
You can specify either or both values for the setting. What you specify and the operating system that you run determines the logic that the Splunk platform uses to locate the certificates. The following table shows the certificate location logic.
caTrustStore valueOperating SystemWhere the Splunk platform looks for CA certificates
splunkLinuxThe location defined by the sslRootCAPath setting
WindowsIf the sslRootCAPathHonoredonWindows setting has a value of "true": The location defined by the sslRootCAPath setting
Otherwise: It does not look for certificates
splunk,OSLinuxThe locations defined by the sslRootCAPath setting and the caTrustStorePath setting
WindowsThe Windows Registry and, if the sslRootCAPathHonoredonWindows setting has a value of "true", the location defined by the sslRootCAPath setting
OSLinuxThe location defined by the caTrustStorePath setting
WindowsThe Windows Registry
If you specify splunk,OS as values, and the Splunk platform finds an identical TLS certificate in both trust stores, it uses the certificate in the Splunk trust store to validate the TLS connection.

Configure the caTrustStorePath setting to determine where the Splunk platform looks for CA certificates

Different Linux distributions use different locations for the CA trust store. The caTrustStorePath setting determines how the Splunk platform looks for the trust store, based on the distribution of Linux you run.

There is support for the setting on machines that run Linux only. It is not applicable for Windows, as Windows stores its certificates in the Windows Registry in all cases except when the sslRootCAPathHonoredonWindows setting has a "true" value.

If caTrustStore has a value of "OS", but this setting has either no value or an invalid value, then the Splunk platform does not attempt to load any certificates from the OS trust store to validate TLS, and logs an error message in the splunkd.log log file.

There is no default value for this setting, which means you must give it one if you give the caTrustStore setting values of either "splunk,OS" or "OS".

Following are example trust store locations for popular Linux distributions:
Linux distributionTrust store location
Debian, Ubuntu, Gentoo: all versions/etc/ssl/certs/ca-certificates.crt
Fedora, Red Hat Enterprise Linux (RHEL): versions 6, 8, and 9/etc/pki/tls/certs/ca-bundle.crt
CentOS, RHEL: version 7/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Prerequisites for certificate usage

There are limitations to the kinds of CA certificates you can reference in the configuration.

Certificates that meet the requirements include:

  • Root certificates with a basicConstraints X.509 V3 extension value of any of the following: "CA:FALSE", "CA:TRUE", or "none".
  • Self-signed CA certificates
  • Certificates that an intermediate CA issues that have a basicConstraints value of "CA:TRUE" as long as all intermediate CA certificates in the certificate chain also have a value of "CA:TRUE"
  • Any certificate chain where all of the certificates in the chain meet all of the previously described requirements

Certificates that do not meet the requirements include:

  • Any root certificate issued by a root CA and whose basicConstraints value has a value of "CA:FALSE"
  • Any certificate or certificate chain that an intermediate CA self-signs