Back up and restore KV store

Back up the KV store and restore it from backup. Taking regular backups from a healthy environment enables you to restore from a backup in the event of a disaster, or if you add a search head to a cluster. You can also take a backup before migrating to a different machine. See Migrate a Splunk Enterprise instance from one physical machine to another in the Installation Manual for more information.

Make sure to be familiar with the standard backup and restore tools and procedures used by your organization.

You can perform different tasks with the KV store, including checking the status, taking a backup, and restoring the KV store to an existing or a new search head or search head cluster. Use the following table to decide which methods to use.

Task Description Limitations
Check the KV store status Before taking a backup or restoring the KV store, you need to check that that KV store is ready. You can also check on backups and restores that are in progress. n/a
Back up and restore the KV store with parallelism Choose this method to process multiple KV store collections simultaneously. This method doesn't guarantee consistency in backup and restore. Changes made during backup aren't always captured.
Backup and restore with point in time consistency Choose this method to guarantee consistency in the backup and restore process. This method captures all changes during the backup process, and blocks all changes during the restore process. You must ensure that all searches, particularly real-time searches, are complete before restoring the KV store. You cannot backup specific apps or collection, only the entire KV store.

This method is not supported with a cohosted KV store.
Backup and restore without guaranteed consistency Choose this method to backup and restore specific apps or collections, or the entire KV store. This method doesn't guarantee consistency in backup and restore. Changes made during backup aren't always captured.

Check the KV store status

To check the status of the KV store, use the show kvstore-status command:

CODE 
./splunk show kvstore-status

The backupRestoreStatus field and the status field indicate the statuses of the KV store. The backupRestoreStatus field indicates the readiness of the node to perform a backup. The status field indicates the status of the storage engine. Both must be in a ready state for you to take a backup.

In a deployment that uses a search head cluster, use the ./splunk show shcluster-status --verbose command at any time to see if any or all cluster members are in maintenance mode. The kvstore_maintenance_status field indicates the captain's status, and the kvstore_status field indicates the status for individual members.

Back up and restore the KV store with parallelism

Use parallel back up and restore to process multiple KV store collections at the same time.

Use parallel back up and restore to process multiple KV store collections at the same time.

You can also measure the parallel backup duration on Splunk Enterprise 10.2 or 10.4 to estimate the time required for the KV store database migration when you upgrade to Splunk Enterprise 10.6.

This backup method does not guarantee point-in-time consistency. Changes made to KV Store data while the backup is running might not be included in the backup. Continue to follow your organization's backup procedures and validate any backup that you plan to use for recovery.

Back up the KV store

Complete the following steps to back up the KV store with parallelism and record the elapsed backup duration.

  1. From the $SPLUNK_HOME/bin directory in the CLI, run the ./splunk show kvstore-status command.

    In a search head cluster deployment, run the backup command on only one cluster member.

  2. Confirm that the status and backupRestoreStatus fields are both in the ready state.

  3. Record the start time of the backup.

  4. Run the following command:
    CODE 
    ./splunk backup kvstore -archiveName <archive_name> -backupParallelJobs true
    Do not include the .tar.gz extension in <archive_name>. Splunk Enterprise creates the archive in the $SPLUNK_DB/kvstorebackup directory and appends the extension.

  5. Monitor the backup by running the ./splunk show kvstore-status command.

  6. When the backupRestoreStatus field returns to the ready state, record the finish time and calculate the parallel backup duration.

Note: Only one KV store backup operation can run at a time. Backup duration can vary based on the number and size of collections, document structure, available CPU resources, disk performance, and concurrent workload.

Prepare to restore the KV store data

Complete the following steps to prepare to restore the KV store data.

  1. Make sure the KV store collection collections.conf file exists on the Splunk Enterprise instance in the same application name that the KV store is going to be restored to.
    CAUTION: If you create the collection collections.conf after restoring the KV store data, the KV store data will be lost
  2. Ensure that your backup archive file is in the $SPLUNK_DB/kvstorebackup directory. In a search head cluster deployment, ensure the file is in this directory on the node from which you are restoring. You only need to restore from one node. The restore replicates across all of the other nodes automatically.
  3. Check that you created the backup archive file from the same collection that you are restoring. You cannot restore a backup to a different collection.

Restore the KV store

Complete the following steps to restore the KV Store with parallelism.
CAUTION: Restoring KV store data overwrites the corresponding KV store data in the destination deployment.
  1. Run the following command:
    CODE 
    ./splunk restore kvstore -archiveName <archive_name>.tar.gz -restoreParallelJobs true
  2. Verify that the restore is complete by running the ./splunk show kvstore-status command:
  3. Confirm that the backupRestoreStatus field has returned to the ready state.

    In a search head cluster deployment, restore the backup from only one cluster member. The restored KV store data replicates to the other cluster members.

Back up and restore the KV store with point in time consistency

Use the following steps to back up the KV store, prepare to restore the KV store data, and then restore the KV store data.

This method is not supported with a cohosted KV store. To see if you have a cohosted KV store, use the splunk show kvstore-status command. If it returns type : Pdl, then you have a cohosted KV store, and you should use the KV store backup with parallelism instead.

Back up the KV store

Complete the following steps to back up the KV store with point in time consistency.

  1. In the CLI, run the splunk show kvstore-status command.
  2. Ensure that the backupRestoreStatus field and the status field are both in the ready state.
  3. If you are running any searches that use outputlookup with the default append=f parameter, end them or allow them to complete before taking a backup, or the backup fails.
  4. (Optional) Create a separate partition for your backup directory, so that the backup is preserved if the $SPLUNK_DB/kvstore directory fails.
  5. Use the splunk enable kvstore-maintenance-mode command to enable maintenance mode. Once you enable maintenance mode, you cannot make any changes to the KV store, and searches that attempt to modify the KV store contents fail. Maintenance mode ensures that the backup completes with consistency.
  6. Use the splunk backup kvstore -pointInTime true command from any search head. This creates an archive file in the $SPLUNK_DB/kvstorebackup directory. You must use the -pointInTime true portion of the command to back up with consistency.
Note: In a search head cluster deployment, only one backup operation can take place at a time. If you initiate a backup on more than one search head at the same time, only one backup succeeds.

To customize your backup, check the full list of arguments for the backup command:

CODE 
./splunk backup kvstore [-pointInTime <true|false>] [-cancel <true|false>] [-parallelCollections <num>] [-archiveName <archive>]
Argument Description
-pointInTime Defaults to false. To take a consistent backup, set it to true.
-cancel Defaults to false. Set the argument to true to cancel an in-progress backup.
-parallelCollections Defaults to 1. Raise the number to increase the number of collections to back up in parallel.
-archiveName Defaults to kvdump_<epoch>.tar.gz. Set to change the name of the backup file. Do not include the extension .tar.gz. It is appended automatically.

Prepare to restore the KV store data

Next, complete the following steps to prepare to restore the KV store data:

  1. Check to see if a backup file was taken with consistency by using the ./splunk show kvstore -archiveName <archive file> command. You can only restore the KV store with consistency with a backup file that was taken with consistency. Backups taken with consistency used the -pointInTime true argument in the backup command.
  2. Make sure the KV store collection collections.conf file exists on the Splunk Enterprise instance in the same application name that the KV store is going to be restored to.
    CAUTION: If you create the collection collections.conf after restoring the KV store data, the KV store data will be lost.
  3. Ensure that your backup archive file is in the $SPLUNK_DB/kvstorebackup directory. If your deployment uses a search head cluster, make sure that the archive file is in that directory on the captain node.
  4. Check that you created the backup archive file from the same collection that you are restoring. You cannot restore a backup to a different collection.

Restore the KV store data to an existing deployment

Now complete the following steps to restore the KV store data.

CAUTION: Restoring KV store data overwrites any KV store data in your Splunk Enterprise instance with the data from the backup.
  1. Ensure all searches are complete, especially real-time searches.
  2. (Optional) To ensure that no searches that use the KV store are started by the scheduler, temporarily disable the scheduler.
  3. If your deployment uses a search head cluster, switch to static captain mode.
  4. Use the splunk enable kvstore-maintenance-mode command to enable maintenance mode. Once you enable maintenance mode, you cannot make any changes to the KV store, and searches that attempt to modify the KV store contents fail. Maintenance mode ensures that the restore completes with consistency.
  5. From the search head cluster captain, restore the KV store data with the splunk restore kvstore -pointInTime true -archiveName <archive> command. Include the .tar.gz extension in the archive name. Even in a clustered deployment, only one restore operation can take place at a time.
  6. Verify that the restore process is complete with the splunk show kvstore-status command.
  7. Disable maintenance mode with the splunk disable kvstore-maintenance-mode command.
  8. If you disabled the scheduler, enable it now.
  9. If your deployment uses a search head cluster, switch back to dynamic captain mode.

To customize your restore, check the full list of arguments for the restore command:

CODE 
./splunk restore kvstore [-pointInTime <true|false>] -archiveName <archive> [-parallelCollection <num>] [-insertionsWorkersPerCollection <num>] [-cancel]
Argument Description
-pointInTime Defaults to false. To restore from a backup taken with consistency, set the argument to true.
-cancel Defaults to false. Set the argument to true to cancel an in-progress restore.
-parallelCollections Defaults to 1. Raise the number to increase the number of collections to restore in parallel, which speeds up the store.
-archiveName Required. Specify the name of the backup file to use. Include the .tar.gz extension in the archive name.
-insertionsWorkersPerCollection Defaults to 1. Raise the number to increase the number of insertion workers per collection, which speeds up the restore.

Back up and restore the KV store without guaranteed consistency

Use the following steps to back up the KV store, prepare to restore the KV store data, and then restore the KV store data, either to an existing deployment or to a new one.

Back up the KV store

Complete the following steps to back up the KV store:

  1. In the CLI, run the splunk show kvstore-status command.
  2. Ensure that the backupRestoreStatus field and the status field are both in the ready state before taking a backup.
  3. (Optional) Create a separate partition for your backup directory, so that the backup is preserved if the $SPLUNK_DB/kvstore directory fails.
  4. To create archive file in the $SPLUNK_DB/kvstorebackup directory, run the splunk backup kvstore command according to the following conditions:
    • If you're backing up a single search head deployment, run the command from the search head.
    • If you're backing up a search head cluster, run the command from the node with the most recent data.
  5. (Optional) Add the following arguments to specify the name of the backup archive file, or define specific collections or apps to back up instead of the entire KV store:
    CODE 
    ./splunk backup kvstore [-archiveName <archive>] [-collectionName <collection>]  [-appName <app>]

Prepare to restore the KV store data

Complete the following steps to prepare to restore the KV store data.

  1. Make sure the KV store collection collections.conf file exists on the Splunk Enterprise instance in the same application name that the KV store is going to be restored to.
    CAUTION: If you create the collection collections.conf after restoring the KV store data, the KV store data will be lost.
  2. Ensure that your backup archive file is in the $SPLUNK_DB/kvstorebackup directory. In a search head cluster deployment, ensure the file is in this directory on the node from which you are restoring. You only need to restore from one node. The restore replicates across all of the other nodes automatically.
  3. Check that you created the backup archive file from the same collection that you are restoring. You cannot restore a backup to a different collection.

Restore the KV store data to an existing deployment

Complete the following steps to restore the KV store data to an existing search head cluster:

CAUTION: Restoring KV store data overwrites any KV store data in your Splunk Enterprise instance with the data from the backup.
  1. Restore the KV store data with the splunk restore kvstore command.
  2. (Optional) Add the following arguments to specify the name of the backup archive file, or specific collections or apps to restore instead of the entire KV store:
    CODE 
    ./splunk restore kvstore [-archiveName <archive>] [-collectionName <collection>]  [-appName <app>]
  3. Verify that the restore process is complete by running the splunk show kvstore-status command.

Restore the KV store data to a new search head cluster

Complete the following steps to create a new search head cluster with new Splunk Enterprise instances. This procedure only works if you took the backup from a search head cluster deployment without using the -pointInTime true argument.

  1. Back up the KV store data from the same search head in the current search head cluster from which you took the backup.
  2. On that search head that will be in the new search head cluster environment, create the KV store collection using the same collection name as the KV store data you are restoring.
  3. Initialize the search head cluster with replication_factor=1
  4. Restore the KV store data to the new search head by using the splunk restore kvstore command.
  5. Run the following command from the CLI: splunk clean kvstore --cluster
  6. Start the Splunk instance and bootstrap with the new search head.
  7. After the KV store has been restored onto the new search head, add the other new search head cluster members.
  8. After complete, change the replication_factor on each search head to the desired replication factor number.
  9. Perform a rolling restart of your deployment.