Upgrade the KV store server version

You must have KV store server version 7.0 or higher before upgrading your deployment to Splunk Enterprise 10.4 or higher. KV store server version 4.2 was deprecated in Splunk Enterprise 9.4, 10.0, and 10.2, and it has been removed from Splunk Enterprise 10.4 and higher. This is a breaking change.

If you are still using KV store server version 4.2, see Upgrade the KV store server version in the Splunk Enterprise 10.0 documentation for instructions on how to upgrade to server version 7.0 or higher before you attempt to upgrade to Splunk Enterprise 10.4 or higher.

As a best practice, continue to upgrade to the highest available KV store server version for continued support and security, and to comply with Splunk Support Policy. For more details, see Splunk Support Policy.

Use the following chart to understand which KV store server versions are available or recommended in each version of Splunk Enterprise:

Splunk Enterprise version KV store server version – lowest available KV store server version – highest available
9.4 4.2 7.0
10.0 4.2 7.0
10.2 4.2 8.0
10.4 7.0 8.0

After your deployment's successful upgrade to Splunk Enterprise 10.2 or higher, Splunk Enterprise automatically upgrades the KV store to server version 8.0. The KV store upgrade is triggered 60 seconds following the first time you start Splunk Enterprise after upgrading your single instance deployment, or after upgrading all of your search head cluster members to the same version of Splunk Enterprise. Splunk Enterprise continues to function while this upgrade is underway.

Work through this process in this order:

  1. Complete the Prerequisites prior to upgrading to Splunk Enterprise version 10.4 or higher, regardless of your intended upgrade path.
  2. Choose one of the following options:
  3. If necessary, Troubleshoot a failed upgrade to the KV store server version.
  4. Complete your upgrade to Splunk Enterprise 10.4 or higher if you have not already done so. For more information about this upgrade, see How to upgrade Splunk Enterprise in the Installation Manual.

Prerequisites

Complete the following checks and preparations before upgrading Splunk Enterprise to ensure your upgrade of Splunk Enterprise and the KV store server version go smoothly.

Check your deployment

Complete the following checks to discover if your deployment supports upgrading the KV store server version.

  1. Ensure you have a supported architecture.
    • For Intel x86_64, a Sandy Bridge or higher Core processor is required, with the SSE4.2, AVX, and AES-NI instructions enabled.
    • For AMD x86_64, a Bulldozer or higher processor is required, with the AVX instructions enabled.
  2. Ensure your system supports Advanced Vector Extensions (AVX). You cannot upgrade to KV store server version 8.0 unless your system supports AVX and you turn it on in your system's CPU settings.

  3. If you have a clustered deployment, ensure that your search head cluster has one of the recommended numbers of healthy members: 3, 5, 7, or any number greater than 7.

  4. If you are using a custom certificate or IPv6 configuration, see Preparing custom certificates for use with KV store to ensure your certificates are configured correctly for compatibility with KV store server version 8.0.
  5. Check whether your certificate is configured with Online Certificate Status Protocol (OCSP) by using the ./splunk cmd openssl x509 -in <path to your certificate> -noout -ocsp_uri command.
    • If this command returns a URL, OSCP is configured, and you must ensure that unencrypted outbound traffic to the OCSP server is permitted. This typically occurs on port 80. If a firewall disallows this traffic from going outbound, set ocspValidation = false in the [kvstore] stanza of your server.conf file and periodically manually download new Certificate Revocation Lists (CRLs), or your upgrade will fail.
    • If this command returns blank, OCSP is not configured. No action is needed.

Prepare for the upgrade

Complete the following steps to prepare your deployment for the upgrade.

  1. You must upgrade to server version 7.0 before upgrading to Splunk Enterprise 10.4 or higher. For instructions and information about updating to KV store server version 7.0, see Upgrade the KV store server version in the Splunk Enterprise 10.0 documentation.
  2. Ensure that more than 50% of your disk space is available.
  3. Confirm that the KV store is healthy by checking its status with the splunk show kvstore-status --verbose command in the command-line interface (CLI). If your KV store is not healthy, file a case using the Splunk Support Portal for help upgrading your deployment. See Support and Services.
  4. If you are using a clustered deployment, ensure that the cluster is healthy before upgrading your deployment. Use the ./splunk show shcluster-status --verbose command in the CLI to confirm the following items:
    • No nodes are in manual detention mode.
    • No nodes are in maintenance mode.
    • No rolling upgrades or restarts are in progress.
    • The captain is stabilized and not frequently switching.
  5. Take a backup of the KV store before initiating your Splunk Enterprise upgrade. For more information about taking a backup of your KV store, see Back up and restore KV store.
    Note: After upgrading Splunk Enterprise but before updating the server version, you do not need to take another backup.

Prepare for a temporary impact on your Splunk Enterprise deployment

The KV store server version upgrade has a short, temporary impact on both the KV store and your overall Splunk Enterprise deployment while the upgrade is ongoing.

The following KV store administrator operations are unavailable during the server version upgrade:

  • KV store maintenance mode
  • Restarting the KV store
  • Resyncing the KV store
  • Backing up or restoring the KV store
  • Any CRUD operations
CAUTION: Do not perform any heavy writes to the KV store during either the automatic or manual upgrade processes. Writes to the KV store performed during upgrade are not saved in the event of a rollback.

If you think you might have a heavy workload on your KV store, see Manage the KV store workload with read-only mode.

Automatically upgrade the KV store server version

Splunk Enterprise automatically upgrades your deployment to server version 8.0 by default when you upgrade to Splunk Enterprise 10.2 and higher. The KV store upgrade begins automatically 60 seconds after your upgrade to Splunk Enterprise completes.

Complete the following steps to verify your server version is upgraded.

  1. Complete any prompts during your Splunk Enterprise upgrade.
  2. Check if your upgrade to the latest KV store server version is in progress or complete with the following CLI command:
    CODE 
    splunk show kvstore-status --verbose
  3. Check the output to see if it indicates the latest server version or an in-progress status. The following output indicates that the KV store is using the latest server version:
    CODE 
    serverVersion : 8.0.x

    The following output indicates that the upgrade is still in progress:

    CODE 
    versionUpgradeInProgress : 1

    If the upgrade is still in progress, check again later.

Manually upgrade the KV store server version

If your KV store server version failed to automatically upgrade to 8.0 during your upgrade to Splunk Enterprise 10.4, you must manually upgrade to server version 8.0 immediately after upgrading to Splunk Enterprise 10.4.

Complete only the following sets of instructions that corresponds to your deployment type: single instance or clustered.

Manually upgrade the KV store server version in a single-instance deployment

Complete the following steps after upgrading Splunk Enterprise.

  1. Check that your instance is ready to migrate with the following CLI command:
    CODE 
    splunk start-standalone-upgrade kvstore -dryRun true
  2. Resolve any issues blocking the upgrade, and then perform the upgrade only if all checks pass.
  3. Initiate the upgrade:
    CODE 
    splunk start-standalone-upgrade kvstore
  4. Track the status of the in-progress upgrade:
    CODE 
    splunk show standalone-kvupgrade-status
  5. Verify that you have the latest version of the KV store server version after the upgrade:
    CODE 
    splunk show kvstore-status --verbose
  6. Check that the output indicates the latest server version:
    CODE 
    serverVersion : 8.0.x

Manually upgrade the KV store server version in a clustered deployment

Complete the following steps after upgrading Splunk Enterprise.

  1. Check that your instance is ready to upgrade with the following CLI command:
    CODE 
    splunk start-shcluster-upgrade kvstore -isDryRun true
  2. Resolve any issues blocking the upgrade, and then perform the upgrade only if all checks pass. Initiate the upgrade only once from any one node. All nodes are automatically upgraded after that.
  3. Initiate the upgrade:
    CODE 
    splunk start-shcluster-upgrade kvstore
  4. Track the status of the in-progress upgrade:
    CODE 
    splunk show shcluster-kvupgrade-status
  5. (Optional) Stop an in-progress upgrade at any time:
    CODE 
    splunk stop-shcluster-upgrade kvstore
  6. Verify that you have the latest version of the KV store server version after the upgrade:
    CODE 
    splunk show kvstore-status --verbose
  7. Check that the output indicates the latest server version:
    CODE 
    serverVersion : 8.0.x

Troubleshoot a failed upgrade to the KV store server version

If your upgrade to KV store server version 8.0 fails, complete the following steps to begin troubleshooting the problem.

Note: While the KV server version upgrade is still in progress, commands that retrieve the status of the KV store upgrade might fail or return a status of "failed" or "ready" intermittently. Retry the command as needed before concluding your upgrade has failed.
  1. Ensure you completed the checks and preparations outlined in the Prerequisites section.
  2. File a case using the Splunk Support Portal for help upgrading your deployment. See Support and Services.
  3. To avoid triggering an attempt at an automatic upgrade every time you restart your Splunk deployment, locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false on all cluster members. While this option is set to false, Splunk Enterprise still reminds you of this pending upgrade every time you restart.

Restart Splunk Enterprise during a server version upgrade

As a best practice, do not restart Splunk Enterprise during the KV store server version upgrade. In Splunk Enterprise 9.4.2 and higher only, the KV store server version upgrade blocks Splunk Enterprise from restarting. If you need to restart during this time anyway, and you are using Splunk Enterprise 9.4.2. or higher, complete the following steps to override this block.

  1. Stop the server version upgrade with one of the following commands:
    • Single instance:
      CODE 
      splunk stop-standalone-upgrade kvstore
    • Clustered deployment:
      CODE 
      splunk stop-shcluster-upgrade kvstore
  2. Verify that the server version upgrade is stopped with one of the following commands:
    • Single instance:
      CODE 
      splunk show standalone-kvupgrade-status
    • Clustered deployment:
      CODE 
      splunk show shcluster-kvupgrade-status
  3. Locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false on all cluster members.
  4. Restart Splunk Enterprise.
CAUTION: If you still cannot restart Splunk Enterprise, you can manually unblock Splunk Enterprise from stopping. However, if you use this command during an in-progress upgrade, data in your KV store might become corrupted or lost. After you ensure that the server version upgrade is stopped, use the following command to manually unblock Splunk Enterprise from restarting: splunk clean-kvstore-upgrade-state.