Manage KV Store collections in Splunk Enterprise Security

Splunk Enterprise Security (ES) uses Splunk KV Store to persist operational, stateful, and enrichment data that supports detections, investigations, and analyst workflows. Managing KV Store storage and retention is critical to maintaining search performance, system stability, and successful upgrades.

Overview of KV Store retention

KV Store retention allows you (the administrator) to control the growth of Splunk Enterprise Security KV Store collections by automatically removing older or surplus records. Without retention policies, collections can grow indefinitely and exceed recommended size limits, leading to degraded performance, instability in search head clusters (SHC), and upgrade failures.

Splunk Enterprise Security supports both time-based and size-based retention policies based on individual collections. In Splunk Enterprise Security version 8.2 and higher, retention can be configured through the UI or configuration files. In Splunk Enterprise Security version 8.3, automated retention is turned on by default for the missioncontrol.findings KV Store collection.

Set up KV Store retention policies for the following reasons:

  • Prevent unbounded data growth.

  • Improve search performance.

  • Reduce upgrade and maintenance risk.

  • Maintain cluster stability.

Best practices when using KV Store retention

Follow these best practices when configuring KV Store retention policies:

  • Turn on retention before KV Store collections approach size limits.

  • Use time-based retention for threat intelligence.

  • Use size-based retention for tracker collections.

  • Align the KV Store retention policies with index retention.

  • Monitor the growth of the KV Store collections regularly.

KV Store architecture in Splunk Enterprise Security

The following criteria must be met by each KV Store collection:
  • Records or rows are identified by a unique key.
  • Optional field types and schemas are enforced.
  • The collection is defined in the collections.confof the app.
  • The collections is located on search heads, not on indexers.
The following criteria must be met by each KV Store collection on search head clusters:
  • Write operations are handled by the cluster captain.
  • KV Store collections are replicated across all cluster members.
  • Optimize the size of KV Store collections since oversized KV Store collections can impact captain elections and cluster health.

Supported KV Store retention controls

Splunk Enterprise Security supports the following retention controls for each KV Store collection:

Control condition Description
Max Age (days) Deletes records that are older than the specified number of days.
Max Size (GB) Deletes the oldest records when the collection exceeds the size limit.
Disabled (-1) Disables the retention control
Note: When both the control conditions such as Max Age and Max Size are configured, either condition can trigger the deletion of the retention policy.

Default limits and recommendations for KV Store collections

Following are some recommended limits and guidance to configure KV Store collections:
Limit Recommendation Notes
Per size collection 25 GB Soft limit; exceeding this can degrade performance
Total KV Store size 100 GB Per search head
Typical retention 7–30 days Varies by collection purpose
Monitoring Required Use SPL or REST endpoints

Expected behavior for retention policies

Following are some expected behaviors for retention policies:
Retention policy Behavior
Retention turned on Oldest records are deleted first
Retention turned off The KV Store collection grows until limits are exceeded
Limit exceeded Writes might fail and searches might slow down