Configure data models for Splunk Enterprise Security

Splunk Enterprise Security leverages data model acceleration to populate dashboards and views and provide detection results. The data models are defined and provided in the Common Information Model add-on (Splunk_SA_CIM), which is included in the Splunk Enterprise Security installation. Splunk Enterprise Security also installs unique data models that only apply to Splunk Enterprise Security content.

Data model acceleration search load

A data model is accelerated through a scheduled summarization search process initiated on the search head. The summarization search runs on the indexers, searching newly indexed data while using the data model as a filter. The resulting matches are saved to disk alongside the index bucket for quick access.

On Splunk platform 9.0 and higher, up to three simultaneous summarization searches can run per data model, per index. To adjust parallel summarization settings on Splunk Cloud Platform, file a support ticket.

Constrain data model searches to specific indexes

The Splunk Common Information Add-on allows you to constrain the indexes searched by a data model for improved performance.

Configure data model acceleration for CIM data models

The Splunk Common Information Add-on allows you to adjust your data model acceleration settings for each data model, including the backfill time, maximum concurrent searches, manual rebuilds, and scheduling priority. If you are using Splunk platform version 9.0 or higher, configure the tags whitelist setting to include any custom tags you use with CIM data models.

Data model acceleration storage and retention

Data model acceleration uses the indexers for processing and storage, placing the accelerated data alongside each index. To calculate the additional storage needed on the indexers based on the total volume of data, use the formula:

Accelerated data model storage per year = Data volume per day * 3.4

You must consider the following assumptions if you use this formula to calculate accelerated data model storage rates:

  • Use the recommended retention rates for the accelerated data models. For example, if you process 100GB/day of data volume for use with Splunk Enterprise Security, you need approximately 340GB of additional space available across all of the indexers to allow for up to one year of data model acceleration and source data retention.
  • Use the default Splunk Enterprise Security data models with the default storage retention rate. Data models outside of that list might require different calculations.
  • The estimated storage is an average of the data models. The estimate might vary significantly if the calculation focuses on only certain data models.
  • The higher the cardinality of the summarized data, the higher the storage requirement, and vice versa.

For more accurate disk estimation, it is best to test with a short data retention period and monitor the disk usage.

Configuring storage volumes

By default, data model acceleration summaries reside in a predefined volume titled _splunk_summaries at the following path: $SPLUNK_DB/<index_name>/datamodel_summary/<bucket_id>/<search_head_or_pool_id>/DM_<datamodel_app>_<datamodel_name> . Data model acceleration storage volumes are managed in indexes.conf using the tstatsHomePath parameter. When configuring new storage volumes, the data model acceleration storage path defaults to the Splunk platform default index path of $SPLUNK_HOME/var/lib/splunk unless explicitly configured otherwise. The storage used for data model acceleration is not added to index sizing calculations for maintenance tasks such as bucket rolling and free space checks.

To manage the data model acceleration storage independently of index settings, you must define a new storage path with [volume:] stanzas.

Data model default retention

The data model retention settings are contingent on the use case and data sources. A shorter retention uses less disk space and requires less processing time to maintain in exchange for limiting the time range of accelerated data.

The following table displays the summary range for data models:

Data model Summary range
AlertsAll Time
Application state1 month
Authentication1 year
Certificates1 year
Change1 year
Change analysis1 year
Compute inventoryAll Time
DLP (Data loss prevention)1 year
DatabasesAll Time
Domain analysis (ES)1 year
Email1 year
Endpoint1 month
Identity managementAll time
Incident management (ES)0
Interprocess messaging1 year
Intrusion detection1 year
InventoryNone
JVM (Java virtual machines)All time
Malware1 year
Network resolution (DNS)3 months
Network sessions3 months
Network traffic3 months
Performance1 month
RiskAll Time
Splunk audit logs1 year
Splunk_CIM_ValidationAll time
Threat intelligence (ES)All time
Ticket management1 year
Updates1 year
Vulnerabilities1 year
Web3 months

You can use the following search to verify the current values: 

| rest splunk_server=local count=0 /services/data/models | table title,acceleration.earliest_time

Use the CIM Setup page in the Splunk Common Information Model app to modify the retention setting for CIM data models. To change the summary range or other settings on a custom data model, manually edit the datamodels.conf provided with the app or add-on. If you are using Splunk Cloud Platform, file a support case to adjust these settings.

Data model acceleration rebuild behavior

In the Splunk platform, if the configuration of the data model structure changes, or the underlying search that creates the data model changes, a complete rebuild of the data model acceleration will initiate. Enterprise Security modifies the default behavior by applying data model configuration changes to the latest accelerations only, and prevents the removal of the prior accelerations. The indexers retain all existing accelerated data models with the prior configuration until the defined retention period is reached, or rolled with the index buckets. Prior configurations for all existing accelerated data models are retained for performance considerations. Rebuilding data models randomly can result in indexers being busy for days at a time. For best performance, do not change the manual rebuilds setting for any data models used by Splunk Enterprise Security.

The rebuild configuration options are managed in the datamodels.conf file.

Data model acceleration enforcement

Splunk Enterprise Security enforces data model acceleration through a modular input.

Follow these steps to disable acceleration for a data model in Splunk Enterprise Security

  1. On the Splunk Enterprise toolbar, open Settings > Data inputs and select Data Model Acceleration Enforcement Settings.
  2. Select a data model.
  3. Uncheck the Acceleration Enforced option.
  4. Save.

Data models used by Splunk Enterprise Security

For reference information about the data models used by Splunk Enterprise Security, see Data models used by ES in the Splunk developer portal.

See also

For more information on parallel summarization, see the product documentation:

Parallel summarization in the Splunk Enterprise Capacity Planning Manual.

For more information on constraining data model searches to specific indexes, see the product documentation:

Set up the Splunk Common Information Model Add-on in the Splunk Common Information Model Add-on User manual.

For more information on accelerating CIM data models, see the product documentation:

For an example of defining a volume and storing data model accelerations, see the product documentation.

For more information on changing the summary range, see the product documentation:

Change the summary range for data model accelerations in the Splunk Common Information Model Add-on User manual.

For more information about acceleration and rebuild behavior, see the product documentation.