Upgrade to a cohosted KV store

Prepare your KV store for upgrade to Splunk Enterprise 10.6 or higher.

Immediately after upgrading to Splunk Enterprise 10.6 or higher, your deployment is automatically upgraded to the new cohosted KV store version 1.0. The cohosted KV store now launches a Storage sidecar, which uses PostgreSQL and runs alongside Splunk Enterprise. For more information about Splunk sidecars, see About Splunk sidecars.

Complete the following steps to prepare for and troubleshoot your upgrade.

Prepare for upgrade

Perform a KV store database migration readiness check before upgrading to Splunk Enterprise 10.6 or higher. Complete the following steps to prepare for upgrade:

  1. Ensure your deployment has KV store server version 7.0 or higher. To check what version of KV store you're using and upgrade if necessary, see Upgrade the KV store server version.

  2. Ensure that more than 50% of your disk space is available.

  3. Confirm that the KV store is healthy by checking its status by using the following command in the CLI:
    CODE 
    splunk show kvstore-status
  4. If you are using a clustered deployment, ensure that the cluster is healthy before upgrading your deployment by using the following command:
    CODE 
    splunk show shcluster-status --verbose
    Confirm the following items in the response:
    • No nodes are in manual detention mode.
    • No nodes are in maintenance mode.
    • No rolling upgrades or restarts are in progress.
    • The captain is stabilized and not frequently switching.
  5. Take a backup of the KV store with parallelism, if you have not done so already. For guidance on taking a parallel backup, see Back up and restore the KV store with parallelism.

  6. Ensure your KV store can be in read-only mode for the duration of the upgrade, which might take an extended time depending on a number of factors:

    • Upgrade time increases with a high number of collections

    • Upgrade time increases with a larger KV store

    • Upgrade time decreases with high number of CPU cores

    • Upgrade time decreases with disk IOPS (Input/output operations per second)

    Note: Premium apps that rely on KV Store writes, such as Enterprise Security and incident reviews or Splunk IT Service Intelligence (ITSI) and glass tables, have degraded functionality during migration.
  7. Prepare for the KV store upgrade to have a temporary impact on both the KV store and your overall Splunk Enterprise deployment while the upgrade is ongoing and the KV store is in read-only mode. The following KV store administrator operations are unavailable during upgrade:
    • KV store maintenance mode
    • Restarting the KV store
    • Resyncing the KV store
    • Backing up or restoring the KV store
    • Any write operations

Initiate and monitor your upgrade

  1. To begin your upgrade to a cohosted KV store version 1.0, upgrade to Splunk Enterprise 10.6 or higher and allow the KV store to upgrade automatically. For more guidance about upgrading Splunk Enterprise, see How to upgrade Splunk Enterprise.

  2. To check the status of your migration to a cohosted KV store, use the following command. Optionally, you can add the --verbose parameter for more information.

    CODE 
    splunk show kvstore-status

    While your upgrade is in progress, this command returns the following information:

    CODE 
    This member:
           backupRestoreStatus : Busy
               migrationStatus : InProgress
            readOnlyPersistent : 0
                        status : readOnly
                 storageEngine : wiredTiger
      versionUpgradeInProgress : 0

    When your upgrade is complete, this command returns the following information:

    CODE 
    This member:
           backupRestoreStatus : Ready
               migrationStatus : Migration_Succeeded
                        status : ready
                 storageEngine : wiredTiger
      versionUpgradeInProgress : 0

Cohosted KVStore Information:
                        status : ready
                          type : Pdl

Optional: Stop or pause an in-progress upgrade to a cohosted KV store

During upgrade to the cohosted KV store, you cannot restart Splunk Enterprise using the CLI. If you need to pause an in-progress upgrade so you can restart Splunk Enterprise, or stop an in-progress upgrade for any other reason, complete the following steps.

  1. Stop the upgrade with the following command:
    CODE 
    splunk stop kvstore-postgres-migration
  2. Verify that the server version upgrade is stopped with the following command:
    CODE 
    splunk show kvstore-postgres-migration-status
Note: If you still cannot restart Splunk Enterprise, clean up the KV store upgrade state to manually unblock restarts. After you ensure that the KV store upgrade is stopped, use the following command to manually unblock Splunk Enterprise from restarting:
CODE 
curl -sku "admin:$SPLUNK_PASSWORD" \
  -X POST \
  'https://127.0.0.1:8089/services/kvstore/migrateToPostgres/cleanup?output_mode=json'
You can also use this command to clean up the upgrade state for any reason.

Optional: Postpone the automatic database upgrade to a cohosted KV store before it begins

Complete the following steps to temporarily postpone the automatic database migration to a cohosted KV store until a later maintenance window.
  1. Before upgrading to Splunk Enterprise 10.6, in Splunk Enterprise 10.4 or 10.2, add the following setting to your local server.conf file.
    CODE 
    [kvstore]
postgresMigrateOnStartup = false
    The default value of postgresMigrateOnStartup is true. Setting it to false prevents Splunk Enterprise from automatically starting the database migration at startup. It does not delete KV store data or permanently cancel the database migration.
  2. In a clustered deployment only: From the deployer, push the bundle, and then wait for the resulting rolling restart to complete.
  3. In a clustered deployment only: Use the following command to confirm the cluster is healthy.
    CODE 
    splunk show shcluster-status --verbose
  4. In a clustered deployment only: Use the following command on every cluster member to verify that postgresMigrateOnStartup= false.
    CODE 
    splunk btool server list kvstore --debug |
       grep postgresMigrateOnStartup
  5. Upgrade to Splunk Enterprise 10.6. If you have a clustered deployment, complete a rolling upgrade to Splunk Enterprise 10.6. After upgrading each member, verify the following:

    • The member is using Splunk Enterprise 10.6.

    • In the member's server.conf file, postgresMigrateOnStartup = false.

    • Logs contain the phrase reason="startup_flag_disabled".

    • When you run a splunk show kvstore-status command, your migrationStatus = NotStarted.

    • The KV is ready and all data remains accessible.

  6. Start Splunk Enterprise 10.6. You should remain on your previous version of KV store.

  7. To allow the automatic database migration during the next maintenance window, set postgresMigrateOnStartup = true on all applicable instances, then restart the instances by following the supported procedure for your deployment architecture.