Manage the KV store workload with read-only mode

These instructions apply only to Splunk Enterprise 9.4.2 and higher.

You can temporarily forcibly stop all writes to the KV store using read-only mode. The most likely case in which you might want to use read-only mode is during the KV store server version upgrade, because heavy writes to the KV store during the upgrade can cause it to fail. For more information about the server version upgrade, see Upgrade the KV store server version.

In a clustered deployment of Splunk Enterprise, you can determine if you need to block writes to the KV store by detecting the KV store's workload. Complete the following steps to determine if you have a high workload, and, if necessary, upgrade a clustered deployment of Splunk Enterprise and the KV store using read-only mode.

  1. Prepare the following SPL command by replacing the placeholder text <search-head-member1>, ..., <search-head-member-n> with your list of search head cluster members.
    index="_introspection" component=KVStoreReplicaSetStats "data.replSetStats.myState"=1 data.replSetStats.set=splunkrs  
host IN (<search-head-member1>, ..., <search-head-member-n>)
| spath data.replSetStats.members{}.name output=searchhead 
| spath data.replSetStats.members{}.stateStr output=state 
| spath data.replSetStats.members{}.optimeDate output=oplogEpoch
| eval prop_key_val=mvzip(mvzip(state, searchhead , "---"), oplogEpoch, "---")
| fields _time, prop_key_val
| eval primary=mvfilter(match(prop_key_val,"^PRIMARY"))
| eval secondary=mvfilter(match(prop_key_val,"^SECONDARY"))
| eval primaryOplogEpoch=mvindex(split(primary,"---"),2)
| fields  _time primaryOplogEpoch, secondary
| fields - _raw
| mvexpand secondary
| rex field=secondary "SECONDARY---(?<searchhead>.*)---(?<oplogEpoch>\d+)"
| eval lagSeconds=(primaryOplogEpoch - oplogEpoch)/1000
| timechart span=1m max(lagSeconds) as MaxLagSeconds by searchhead limit=0
  2. Run your modified command in the Splunk Enterprise Search & Reporting app from a deployer that has access to introspection logs on all search head cluster members.
  3. Explore the results of your command. If the results show that the lag between the primary and secondary KV store replies is consistently above 10 seconds, then your workload is high.
    If these commands confirm that you have a high workload, then use read-only mode to block heavy writes to the KV store while you upgrade the KV store server version. This prevents the upgrade from failing due to heavy writes.
Upgrade the KV store server version in a clustered deployment with read-only mode.
CAUTION: Writes to the KV store performed during an upgrade of Splunk Enterprise or the KV store server version are not necessarily saved in the event of a rollback. To prevent failure of the KV store server version upgrade, you can choose to completely block writes to the KV store during the upgrade by using read-only mode.
  1. Turn off the automatic upgrade of the KV store. Before initiating the upgrade to Splunk Enterprise 9.4.x, locate the [kvstore] stanza of the server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false.
  2. Upgrade to Splunk Enterprise 9.4.x.
    For more information about completing this upgrade, see How to upgrade Splunk Enterprise.
  3. Switch to a static KV store captain for your search head cluster.
    For more information about switching to a static captain, see Switch to a static captain.
  4. Turn on read-only mode on the captain by entering the following line in the command-line interface (CLI).
    splunk enable kvstore-maintenance-mode -readOnly true
  5. Upgrade the KV store server version manually, monitoring its progress until it is complete.
    For more information about manual upgrade, see Manually upgrade the KV store server version in a clustered deployment.
  6. Turn off read-only mode with the following command in the CLI.
    splunk disable kvstore-maintenance-mode -readOnly true
  7. Revert to a dynamic captain for the search head cluster.
    For more information about switching back to a dynamic captain, see Revert to the dynamic captain.

After completing these steps, your Splunk Enterprise and KV store server version upgrades are complete.

To troubleshoot this process or learn more about it, see Upgrade the KV store server version

You can turn on or off read-only mode at any time by entering the following commands into the CLI. If you have a clustered deployment, enter these commands from the KV store captain.

To turn on read-only mode, use the following command.
splunk enable kvstore-maintenance-mode -readOnly true
To turn off read-only mode, use the following command.
splunk disable kvstore-maintenance-mode -readOnly true