Upgrade the KV store server version

Splunk Enterprise versions 9.4 and higher no longer support KV store server version 4.2. Upgrade to KV store server version 7.0 for continued support and security, and to comply with Splunk Support Policy. For more details, see Splunk Support Policy.

After your deployment's successful upgrade to Splunk Enterprise 9.4 or higher, Splunk Enterprise automatically upgrades the KV store to server version 7.0. The KV store upgrade is triggered 60 seconds following the first time you start Splunk Enterprise after upgrading your single instance deployment, or after upgrading all of your search head cluster members to the same version of Splunk Enterprise. Splunk Enterprise continues to function while this upgrade is underway.

Note: As an alternative, you can configure your deployment prior to the upgrade to prevent the automatic server version upgrade. If you choose this upgrade path, then immediately upgrade the server version manually after completing your upgrade to Splunk Enterprise 9.4.

Work through this process in this order:

  1. Complete the Prerequisites prior to upgrading to Splunk Enterprise version 9.4.x or higher, regardless of your intended upgrade path.
  2. Choose one of the following options:
  3. If necessary, Troubleshoot a failed upgrade to server version 7.0.
  4. Complete your upgrade to Splunk Enterprise 9.4.x if you have not already done so. For more information about this upgrade, see How to upgrade Splunk Enterprise in the Installation Manual.

Prerequisites

Complete the following checks and preparations before upgrading Splunk Enterprise to ensure your upgrade of Splunk Enterprise and the KV store server version go smoothly.

Check your deployment

Complete the following checks to discover if your deployment supports upgrading the KV store server version.

  1. Ensure you have a supported architecture.
    • For Intel x86_64, a Sandy Bridge or higher Core processor is required, with the SSE4.2, AVX, and AES-NI instructions enabled.
    • For AMD x86_64, a Bulldozer or higher processor is required, with the AVX instructions enabled.
  2. Ensure your system supports Advanced Vector Extensions (AVX). You cannot upgrade to KV store server version 7.0 unless your system supports AVX and you turn it on in your system's CPU settings.
  3. Splunk Enterprise 9.4.2 and higher only: If you are using a custom certificate or IPv6 configuration, see Preparing custom certificates for use with KV store to ensure your certificates are configured correctly for compatibility with KV store server version 7.0.
  4. Splunk Enterprise 9.4.1 and lower only: Check your server.conf file to ensure that you aren't using a custom certificate for the KV store. Compare with the following example: 
    [kvstore]
sslVerifyServerCert = true
sslVerifyServerName = true
caCertFile = $SPLUNK_HOME/<path_to_ca_pem>
serverCert = $SPLUNK_HOME/<path_to_server_pem>
sslPassword = <PASSWORD>

    If you are using a custom certificate, upgrading to server version 7.0 is not currently supported. Upgrades for KV store deployments with custom certificates are available in Splunk Enterprise 9.4.2 or higher only. To work around this issue, you can revert to a default certificate.

  5. Splunk Enterprise 9.4.1 and lower only: Check your server.conf file to ensure that you aren't using an IPv6 configuration. Compare with the following example: 
    [general]
listenOnIPv6 = only / yes

    If you are using an IPv6 configuration, upgrading to server version 7.0 is not currently supported. To work around this issue, you can turn off your IPv6 configuration or use Splunk Enterprise 9.4.2 or higher instead.

Prepare for upgrade

Complete the following steps to prepare your deployment for upgrade.

  1. You must upgrade to server version 4.2.x before upgrading to Splunk Enterprise 9.4.x or higher. For instructions and information about updating to KV store server version 4.2.x in Splunk Enterprise versions 9.0.x through 9.3.x, see Migrate the KV store storage engine in the Splunk Enterprise 9.3.0 documentation.
  2. Ensure that more than 50% of your disk space is available.
  3. Confirm that the KV store is healthy by checking its status with the splunk show kvstore-status --verbose command in the CLI. If your KV store is not healthy, locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false, then file a case using the Splunk Support Portal for help upgrading your deployment. See Support and Services.
  4. If you are using a clustered deployment, ensure that the cluster is healthy before upgrading your deployment. Use the ./splunk show shcluster-status --verbose command in the CLI to confirm the following items:
    • No nodes are in manual detection mode.
    • No nodes are in maintenance mode.
    • No rolling upgrades or restarts are in progress.
    • The captain is stabilized and not frequently switching.
  5. Take a backup of the KV store before initiating your Splunk Enterprise upgrade. For more information about taking a backup of your KV store, see Back up and restore KV store.
    Note: After upgrading Splunk Enterprise but before updating the server version, it is not necessary to take another backup. Splunk Enterprise automatically takes a backup of the KV store at this time. If the KV store server version upgrade fails, Splunk Enterprise automatically restores the KV store from the last backup taken before server version upgrade.

Prepare for a temporary impact on your Splunk Enterprise deployment

The KV store server version upgrade has a temporary impact on both the KV store and your overall Splunk Enterprise deployment while the upgrade is ongoing.

The following KV store administrator operations are unavailable during the server version upgrade:

  • KV store maintenance mode
  • Restarting the KV store
  • Resyncing the KV store
  • Backing up or restoring the KV store
  • Any CRUD operations
CAUTION: Do not perform any heavy writes to the KV store during either the automatic or manual upgrade processes. Writes to the KV store performed during upgrade are not saved in the event of a rollback.

Splunk Enterprise 9.4.2 and higher only: If you think you might have a heavy workload on your KV store, see Manage the KV store workload with read-only mode.

Automatically upgrade to KV store server version 7.0

If you are using KV store server version 4.2.x at the time of your upgrade to Splunk Enterprise 9.4 or higher, then Splunk Enterprise automatically upgrades your deployment to server version 7.0 by default. Complete the following steps to verify your server version is upgraded.

  1. Complete any prompts during your Splunk Enterprise upgrade.
  2. Verify that you have the latest version of the KV store server version after the upgrade with the following command, either in the command-line interface (CLI) or through the REST API:
    CLI: 
    splunk show kvstore-status --verbose

    REST: 

    curl -k -u admin:changeme https://localhost:8089/services/kvstore/version
  3. Check the output to see that it indicates the latest server version:
    CLI: 
    serverVersion : 7.0.14

    REST: 

    <s:key name="version">7.0.14</s:key>
The following output indicates that the upgrade is still in progress, from both CLI or REST: 
versionUpgradeInProgress : 1
If the upgrade is still in progress, check again later.

Manually upgrade to KV store server version 7.0

If you don't want your KV store server version to automatically upgrade to 7.0 at the same time as you upgrade to Splunk Enterprise 9.4.x, you can turn off the automatic upgrade. If you choose this upgrade path, you must manually upgrade to server version 7.0 immediately after upgrading to Splunk Enterprise 9.4.x.

Complete the following steps to prepare to manually upgrade your KV store server version.

  1. Before initiating the upgrade to Splunk Enterprise 9.4.x, locate the [kvstore] stanza of the server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false.
  2. Determine your deployment type. If your single instance of the KV store is located on a search head, the cluster manager, or any indexer node, you have a single-instance KV store deployment. If you have multiple KV store nodes across a search head cluster, then you have a clustered KV store deployment.
  3. Upgrade to Splunk Enterprise 9.4.x. For more information about completing this upgrade, see How to upgrade Splunk Enterprise.
  4. Complete the steps in one of the two following sections, choosing according to your deployment type.

Manually upgrade the KV store server version in a single-instance deployment

Complete the following steps after upgrading to Splunk Enterprise 9.4.x.

  1. Check that your instance is ready to migrate with one of the following commands, either in the CLI or through the REST API:
    CLI: 
    splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/kvstore/version/upgrade -d version=7.0 -d dryRun=true
  2. Resolve any issues blocking the upgrade, and then perform the upgrade only if all checks pass.
  3. Use one of the following commands to initiate this upgrade:
    CLI: 
    splunk start-standalone-upgrade kvstore -version 7.0

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/kvstore/version/upgrade -d version=7.0
  4. Track the status of the in-progress upgrade with one of the following commands:
    CLI: 
    splunk show standalone-kvupgrade-status

    REST: 

    curl -ku admin:changeme https://localhost:8089/services/kvstore/version
  5. Verify that you have the latest version of the KV store server version after upgrade with one of the following commands:
    CLI: 
    splunk show kvstore-status --verbose

    REST: 

    curl -k -u admin:changeme https://localhost:8089/services/kvstore/version
  6. Check that the output indicates the latest server version:
    CLI: 
    serverVersion : 7.0.14

    REST: 

    <s:key name="version">7.0.14</s:key>

Manually upgrade the KV store server version in a clustered deployment

Complete the following steps after upgrading to Splunk Enterprise 9.4.x.

  1. Check that your instance is ready to upgrade with one of the following commands, either in the CLI or through the REST API:
    CLI: 
    splunk start-shcluster-upgrade kvstore -version 7.0 -isDryRun true

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvstore-upgrade/start -d version=7.0 -d isDryRun=true
  2. Resolve any issues blocking the upgrade, and then perform the upgrade only if all checks pass. Initiate the upgrade only once from any one node. All nodes are automatically upgraded after that.
  3. Use one of the following commands to initiate this upgrade:
    CLI: 
    splunk start-shcluster-upgrade kvstore -version 7.0

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvstore-upgrade/start -d version=7.0
  4. Track the status of the in-progress upgrade with one of the following commands:
    CLI: 
    splunk show kvstore-upgrade-status

    REST: 

    curl -ku admin:changeme https://localhost:8089/services/shcluster/captain/kvstore-upgrade/status
  5. (Optional) Stop an in-progress upgrade at any time with the following REST API command: 
    curl -X POST -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvstore-upgrade/stop
  6. Verify that you have the latest version of the KV store server version after the upgrade with one of the following commands:
    CLI: 
    splunk show kvstore-status --verbose

    REST: 

    curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvstore-upgrade/status
  7. Check that the output indicates the latest server version:
    CLI: 
    serverVersion: 7.0.14

    REST: 

    version: 7.0.14

Troubleshoot a failed upgrade to server version 7.0

If your upgrade to KV store server version 7.0 fails, complete the following steps to begin troubleshooting the problem.

  1. Ensure you completed the checks and preparations outlined in the Prerequisites section.
  2. File a case using the Splunk Support Portal for help upgrading your deployment. See Support and Services.
  3. To avoid triggering an attempt at an automatic upgrade every time your restart your Splunk deployment, locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false. While this option is set to false, Splunk Enterprise still reminds you of this pending upgrade every time you restart.

Restart Splunk Enterprise during a server version upgrade

As a best practice, do not restart Splunk Enterprise during the KV store server version upgrade. In Splunk Enterprise 9.4.2 and higher only, the KV store server version upgrade blocks Splunk Enterprise from restarting. If you need to restart during this time anyway, and you are using Splunk Enterprise 9.4.2. or higher, complete the following steps to override this block.

  1. Stop the server version upgrade with one of the following commands:
    • Single instance: 
      splunk stop-standalone-upgrade kvstore
    • Clustered deployment: 
      splunk stop-shcluster-upgrade kvstore
  2. Verify that the server version upgrade is stopped with one of the following commands:
    • Single instance: 
      splunk show standalone-kvupgrade-status
    • Clustered deployment: 
      splunk show shcluster-kvupgrade-status
  3. Locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false on all cluster members.

  4. Restart Splunk Enterprise

CAUTION: If you still cannot restart Splunk Enterprise, you can manually unblock Splunk Enterprise from stopping. However, if you use this command during an in-progress upgrade, data in your KV store might become corrupted or lost. After you ensure that the server version upgrade is stopped, use the following command to manually unblock Splunk Enterprise from restarting: splunk clean-kvstore-upgrade-state.