How to prepare TLS certificates for use with the Splunk platform

TLS certificates let you secure communication between Splunk Enterprise components from end to end. After you get the certificates, you need to prepare them for use with your deployment before you install them and configure your deployment to use them.

As part of preparing TLS certificates for use, you must combine them with the private keys that you either received or generated to create the certificates into a single certificate file that the Splunk platform can use.

All certificates and respective keys that you use on the Splunk platform must be concatenated in this manner. A certificate or key alone does not work, even if you configure it correctly. Regardless of the service or contact point you secure, they all must use combined certificates files.

Create a single combined certificate file

After you obtain certificates, several files are available depending on the method you used to get them. You will combine these files into one file. You must combine the files in the right order and the combined file must be in the correct format. If you don't combine them correctly, your Splunk platform instance won't be able to use the file to secure its communications with other instances.

If you got the certificate by purchasing them from a certificate authority, you'll have the following at a minimum:

  • The private key file
  • The server certificate file
  • The certificate authority certificate file, which was used to create the server certificate

Note: The certificate authority certificate is also known as the root certificate.

If you got the certificate by creating a certificate signing request and submitting that request to a CA, you will have the following:

  • The private key file that you created and subsequently used to create the certificate signing request
  • The certificate signing request file
  • The server certificate file that you downloaded from the certificate authority after submitting your certificate signing request.
  • The certificate authority certificate file that you downloaded from the certificate authority after downloading the server certificate.

If you created and signed a certificate yourself, you will have the following:

  • The private key file that you used to create and sign the certificate authority certificate.
  • The certificate authority certificate signing request file
  • The root certificate file that you generated with the private key file and the certificate authority certificate signing request file
  • The private key file that you created to create and sign the server certificate
  • The server certificate signing request file
  • The server certificate file. You created this file using the private key and the server certificate signing request file

Depending on the method you used, you must combine the server certificate, the private key, and the public certificate, in that order, into a single file. The combined file must be in privacy-enhanced mail (PEM) format.

*nix command Windows command
CODE 
cat <server certificate file> <server private key file> > <combined server certificate file>
CODE 
type <server certificate file> <server private key file> > <combined server certificate file>

After you create the combined certificate file, review it using a text editor. Its contents must contain, in the following order:

  • The server certificate
  • The private key

Following is an example of a properly concatenated certificate. Each certificate and key must include the "BEGIN" and "END" markers to be considered complete.

CODE 
-----BEGIN CERTIFICATE-----
        MIICUTCCAboCCQCscBkn/xey1TANBgkqhkiG9w0BAQUFADBtMQswCQYDVQQGEwJV
        ...
        <Server Certificate>
        ...
        8/PZr3EuXYk1c+N5hgIQys5a/HIn
        -----END CERTIFICATE-----
        -----BEGIN RSA PRIVATE KEY-----
        Proc-Type: 4,ENCRYPTED
        DEK-Info: DES-EDE3-CBC,CFCECC7976725DE5
        
        S+DPcQ0l2Z1bk71N3cBqr/nwEXPNDQ4uqtecCd3iGMV3B/WSOWAQxcWzhe9JnIsl
        ...
        <Server Private Key – Passphrase protected>
        ...
        -----END RSA PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        MIICUTCCAboCCQCscBkn/xey1TANBgkqhkiG9w0BAQUFADBtMQswCQYDVQQGEwJV
        ...
        <Certificate Authority Public Key>
        ...
        8/PZr3EuXYk1c+N5hgIQys5a/HIn

        -----END CERTIFICATE-----

How to configure a certificate chain

To use multiple certificates, place any intermediate certificates after the server certificate. You can add as many intermediate certificates as you need, in decreasing order of hierarchy.

Concatenate multiple certificates in the following order:

CODE 
[ server certificate]
[ intermediate certificate]
[ certificate authority certificate (if required) ]

The following is an example of a certificate chain:

CODE 
-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----

Note: Valid certificate chains must contain the -----BEGIN CERTIFICATE---- and -----END CERTIFICATE----- certificate markers to be valid. Do not remove these markers from the certificate file.

In another example, when you use Splunk forwarder to indexer certificates that contain a Private Key, the completed certificate file might look like the following:

CODE 
-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
...<Server Private Key – Passphrase protected>
 -----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
... (certificate for your server)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... (the intermediate certificate)...
-----END CERTIFICATE-----

Set correct permissions on TLS certificate files

When you install custom TLS certificates, confirm that their file system permissions are at the right level of restriction.

On *nix operating systems, certificate files must be readable and writable by the user that runs Splunk processes on the machine, with no other users having access (the equivalent of chmod 0600). Some Splunk platform services, including App Key Value Store (KV Store), require restrictive certificate file permissions and will not start if permissions are too open.

On Windows, grant only the account that runs the Splunk daemon Read permission on the certificate file, and remove access for all other users and groups.

Set correct permissions on TLS certificate files

On *nix systems, run the following command to set the correct permissions on your certificate file. Replace /opt/splunk/etc/auth/mycerts/server.pem with the path to your certificate file:

BASH 
chmod 0600 /opt/splunk/etc/auth/mycerts/server.pem

On Windows systems, use the icacls command line utility to remove inherited permissions and grant only the Splunk service account Read access. Replace C:\Program Files\Splunk\etc\auth\mycerts\server.pem with the path to your certificate file, and replace SPLUNK_SERVICE_ACCOUNT with the name of the account that runs the Splunk service:

BASH 
icacls "C:\Program Files\Splunk\etc\auth\mycerts\server.pem" /inheritance:r /grant:r "SPLUNK_SERVICE_ACCOUNT:(R)"

After changing permissions, restart the Splunk platform:

BASH 
$SPLUNK_HOME/bin/splunk restart

Certificate file permissions error message

The following error message appears in the splunkd.log log file when certificate file permissions are more open than chmod 0640. On Windows, the equivalent error occurs when the certificate file grants Read or Write access to users or groups beyond the Splunk service account. You might see an error similar to the following:

CODE 
FATAL: private key file "/opt/splunk/etc/auth/mycerts/server.pem" has group or world access
DETAIL: File must have permissions u=rw (0600) or less if owned by the database user,
or permissions u=rw,g=r (0640) or less if owned by root.

The default permissions applied to Splunk platform certificate files can be wider than this, so you might encounter this error after placing custom certificates without explicitly setting permissions.

Next steps

After you combine certificates into one file, you can then configure the Splunk platform to use the certificates.