Manage KV Store collections in Splunk Enterprise Security

Splunk Enterprise Security (ES) uses Splunk KV Store to persist operational, stateful, and enrichment data that supports detections, investigations, and analyst workflows. Managing KV Store storage and retention controls is critical to maintaining search performance, system stability, and successful upgrades.

Overview of KV Store retention controls

KV Store retention controls allow you (the administrator) to control the growth of Splunk Enterprise Security KV Store collections by automatically removing older or surplus records. Without retention controls, collections can grow indefinitely and exceed recommended size limits, leading to degraded performance, instability in search head clusters (SHC), and upgrade failures.

Splunk Enterprise Security supports both time-based and size-based retention controls based on individual collections. In Splunk Enterprise Security version 8.2 and higher, retention controls can be configured through the UI or configuration files. In Splunk Enterprise Security version 8.3, automated retention controls are turned on by default for the missioncontrol.findings KV Store collection.

Set up KV Store retention controls for the following reasons:

  • Prevent unbounded data growth.

  • Improve search performance.

  • Reduce upgrade and maintenance risk.

  • Maintain cluster stability.

Best practices when using KV Store retention controls

Follow these best practices when configuring KV Store retention controls:

  • Turn on retention controls before KV Store collections approach size limits.

  • Use time-based retention controls for threat intelligence.

  • Use size-based retention controls for tracker collections.

  • Align the KV Store retention controls with index retention policies.

  • Monitor the growth of the KV Store collections regularly.

KV Store architecture in Splunk Enterprise Security

The following criteria must be met by each KV Store collection:
  • Records or rows are identified by a unique key.
  • Optional field types and schemas are enforced.
  • The collection is defined in the collections.confof the app.
  • The collections is located on search heads, not on indexers.
The following criteria must be met by each KV Store collection on search head clusters:
  • Write operations are handled by the cluster captain.
  • KV Store collections are replicated across all cluster members.
  • Optimize the size of KV Store collections since oversized KV Store collections can impact captain elections and cluster health.

Supported KV Store retention controls

Splunk Enterprise Security supports the following retention controls for each KV Store collection:

Control condition Description
Max Age (days) Deletes records that are older than the specified number of days.
Max Size (GB) Deletes the oldest records when the collection exceeds the size limit.
Disabled (-1) Turns off the retention control
Note: When both the control conditions such as Max Age and Max Size are configured, retention controls are enforced only after both conditions are met. Data is not eligible for removal unless it satisfies both the age and size thresholds.

Default limits and recommendations for KV Store collections

Following are some recommended limits and guidance to configure KV Store collections:
Limit Recommendation Notes
Per size collection 25 GB Soft limit; exceeding this can degrade performance
Total KV Store size 100 GB Per search head
Typical retention control 7–30 days Varies by collection purpose
Monitoring Required Use SPL or REST endpoints

Expected behavior for retention controls

Following are some expected behaviors for retention controls:
Retention control policy Behavior
Retention control turned on Oldest records are deleted first
Retention control turned off The KV Store collection grows until limits are exceeded
Limit exceeded Writes might fail and searches might slow down