Preparing custom certificates for use with KV store

These instructions apply only to Splunk Enterprise 9.4.2 and higher.

For leaf and Certificate Authority (CA) certificates to work in KV store server version 7.0, you must meet the following requirements. Otherwise, you cannot upgrade your KV store to server version 7.0. If you don't meet these requirements at any time after upgrade, your KV store service might present errors or fail.

KV store server version 7.0 certificate requirements

  • Your KV store certificate must not contain any expired leaf or CA certificates. If your CA certificates are expired, remove them. If your leaf certificate is expired, generate a new one. To check if your leaf or CA certificates are expired, use one of the following commands in the command-line interface (CLI).

    • In Windows, copy each certificate block into it's own file, and then enter the following command for each file.
      $SPLUNK_HOME\bin\splunk.exe cmd openssl -checkend 0 -in <one file at a time>
    • In Unix, base your command on the following example command, adjusting the file paths as needed. Alternatively, you can base your command on the previous Windows command, checking each file one at a time.
      awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if (/BEGIN CERTIFICATE/){c++} cert = cert $0 "\n"; if (/END CERTIFICATE/) { printf "Checking Cert #%d...", c; if (system("echo \"" cert "\" | openssl x509 -checkend 0 -noout >/dev/null 2>&1") != 0) { system("echo \" -> EXPIRED\" && echo \"" cert "\" | $SPLUNK_HOME/bin/splunk openssl x509 -noout -subject -enddate"); expired=1 } else { print " -> OK" } cert="" } } END { if (expired==1) { print "\nFound one or more expired certificates." } else { print "\nAll certificates are valid." } }' <file path to serverCert>

  • Your KV store certificate must have both client (SSL client : Yes) and server (SSL server : Yes) purpose. If your leaf certificate does not contain these values, regenerate the certificate and set extendedKeyUsage = clientAuth, serverAuth. To check if your certificate meets these requirements, use one of the following commands:

    • Windows
      $SPLUNK_HOME\bin\splunk.exe cmd openssl x509 -noout -text -purpose -in <file path to serverCert>
    • Unix
      $SPLUNK_HOME/bin/splunk cmd openssl x509 -noout -text -purpose -in <file path to serverCert>

  • Your sslRootCAPath or caCertFile file must not contain any expired CA certificates. If you don't know which of these files you're using, first complete the steps in the "Prepare and verify certificates" section to identify which one. If you do know which of these files you're using, check it for expired CA certificates by entering one of the following commands into the CLI.

    • In Windows, copy each certificate block into it's own file, and then enter the following command for each file.
      $SPLUNK_HOME\bin\splunk.exe cmd openssl -checkend 0 -in <one file at a time>
    • In Unix, base your command on the following example command, adjusting the file paths as needed. Alternatively, you can base your command on the previous Windows command, checking each file one at a time.
      awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if (/BEGIN CERTIFICATE/){c++} cert = cert $0 "\n"; if (/END CERTIFICATE/) { printf "Checking Cert #%d...", c; if (system("echo \"" cert "\" | openssl x509 -checkend 0 -noout >/dev/null 2>&1") != 0) { system("echo \" -> EXPIRED\" && echo \"" cert "\" | $SPLUNK_HOME/bin/splunk openssl x509 -noout -subject -enddate"); expired=1 } else { print " -> OK" } cert="" } } END { if (expired==1) { print "\nFound one or more expired certificates." } else { print "\nAll certificates are valid." } }' <file path to sslRootCAPath or caCertFile>

  • Your sslRootCAPath and caCertFile directories must contain the full issuer chain used by the KV store certificate, including all intermediate and root CA certificates. If you use a KV store cluster, all members of the cluster must have the CAs of all other peers' KV store certificates.

Prepare and verify certificates

Complete the following steps to prepare and verify your certificates. This process prevents certificate-related issues after your upgrade to Splunk Enterprise 9.4.2 or higher.

  1. To identify which certificates your KV store is currently using, enter one of the following commands in the command-line interface (CLI).
    OptionDescription
    Unix 
    $SPLUNK_HOME/bin/splunk cmd btool server list kvstore
    Windows 
    $SPLUNK_HOME\bin\splunk.exe cmd btool server list kvstore
  2. In the command's output, if sslVerifyServerCert is set to true, then note down the serverCert and caCertFile values.
    However, if sslVerifyServerCert is not present or it's set to false, discard the previous command's output, and use the commands in Step 3 instead.
  3. If the output from Step 1 showed sslVerifyServerCert is set to false, then enter one of the following commands in the CLI. Otherwise, the following commands do not apply to you. Skip this step, and and go directly to Step 4.

    If you are using a FIPS or Common Criteria operating system, the following commands do not apply to you. Skip this step, and and go directly to Step 4.

    OptionDescription
    Unix 
    $SPLUNK_HOME/bin/splunk cmd btool server list sslConfig
    Windows 
    $SPLUNK_HOME\bin\splunk.exe cmd btool server list sslConfig
  4. From whichever command was appropriate for your use case, make a note of the file path for the serverCert and sslRootCAPath parameters.
    If sslRootCAPath is not present, you can note down the file path after caCertFile instead in its place. This option is deprecated. sslRootCAPath takes precedence over caCertFile. Do not use caTrustStore.
  5. Using the file paths you noted in the previous step, enter the following command into the CLI:
    OptionDescription
    Unix 
    $SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
    Windows 
    $SPLUNK_HOME\bin\splunk.exe cmd openssl verify -verbose -x509_strict -CAfile <file path to sslRootCAPath or caCertFile> <file path to serverCert>
  6. To verify that your certificate is properly signed, ensure the output from the previous step matches the following example response, where instead of this example file path, it shows the file path to your server certificate: 
    $SPLUNK_HOME/etc/auth/server.pem: OK

If you see any output other than OK, you are missing one or more CA certificates from your sslRootCAPath or caCertFile file. You must locate these certificates and append them to the existing sslRootCAPath or caCertFile file, then run the command again until you receive an OK status.

Note: The sslRootCAPath file could be located on a deployment server, a search head cluster member, a cluster manager, or a third party server. If you edit this file, ensure you edit it in the correct location for your deployment.

See the following documentation for more information about troubleshooting your certificates: