Accelerate data models

Data model acceleration is a tool that you can use to speed up data models that represent extremely large datasets. After acceleration, pivots based on accelerated data model datasets complete quicker than they did before, as do reports and dashboard panels that are based on those pivots.

Data model acceleration does this with the help of the High Performance Analytics Store functionality, which builds data summaries behind the scenes in a manner similar to that of report acceleration. Like report acceleration summaries, data model acceleration summaries are easy to turn on and turn off. Data model acceleration summaries are created on the indexer tier and are associated with the buckets of their source indexes. Their storage location depends on whether the source index uses SmartStore.

This topic covers:

  • The differences between data model acceleration, report acceleration, and summary indexing.
  • How you turn on persistent acceleration for data models.
  • How Splunk software builds data model acceleration summaries.
  • How you can search accelerated data model acceleration summaries with the tstats command.
  • Advanced configurations for persistently accelerated data models.

This topic also explains ad hoc data model acceleration. Splunk software applies ad hoc data model acceleration whenever you build a pivot with an unaccelerated dataset. It is even applied to transaction-based datasets and search-based datasets that use transforming commands, which can't be accelerated in a persistent fashion. However, any acceleration benefits you obtain are lost the moment you leave the Pivot Editor or switch datasets during a session with the Pivot Editor. These disadvantages do not apply to "persistently" accelerated datasets, which will always load with acceleration whenever they're accessed via Pivot. In addition, unlike "persistent" data model acceleration, ad hoc acceleration is not applied to reports or dashboard panels built with Pivot.

How data model acceleration differs from report acceleration and summary indexing

This is how data model acceleration differs from report acceleration and summary indexing:

  • Report acceleration and summary indexing speed up individual searches, on a report by report basis. They do this by building collections of precomputed search result aggregates.
  • Data model acceleration speeds up reporting for the entire set of fields that you define in a data model and which you and your Pivot users want to report on. In effect it accelerates the dataset represented by that collection of fields rather than a particular search against that dataset.

What is a high-performance analytics store?

Data model acceleration summaries are composed of multiple time-series index files, which have the .tsidx file extension. Each .tsidx file contains records of the indexed field::value combos in the selected dataset and all of the index locations of those field::value combos. It's these .tsidx files that make up the high-performance analytics store. Collectively, the .tsidx files are optimized to accelerate a range of analytical searches involving the set of fields defined in the accelerated data model.

An accelerated data model's high-performance analytics store spans a summary range. This is a range of time that you select when you activate acceleration for the data model. When you run a pivot on an accelerated dataset, the pivot's time range must fall at least partly within this summary range in order to get an acceleration benefit. For example, if you have a data model that accelerates the last month of data but you create a pivot using one of this data model's dataset that runs over the past year, the pivot will initially only get acceleration benefits for the portion of the search that runs over the past month.

The .tsidx files that make up a high-performance analytics store for a single data model are always distributed across one or more of your indexers. This is because Splunk software creates .tsidx files on the indexer, parallel to the buckets that contain the events referenced in the file and which cover the range of time that the summary spans.

Note: The high-performance analytics store created through persistent data model acceleration is different from the summaries created through ad hoc data model acceleration. Ad hoc summaries are always created in a dispatch directory at the search head.

See About ad hoc data model acceleration.

Turn on persistent acceleration for a data model

See Managing Data Models to learn how to activate data model acceleration.

Data model acceleration limitations

There are a number of restrictions on the kinds of data model datasets that can be accelerated.

  • Datasets can be accelerated only if they contain at least one root event hierarchy or one root search hierarchy that only includes streaming commands. Dataset hierarchies based on root search datasets that include nonstreaming commands and root transaction datasets are not accelerated.
    • Pivots that use unaccelerated datasets fall back to _raw data, which means that they initially run more slowly. However, they can receive some acceleration benefit from ad hoc data model acceleration. See About ad hoc data model acceleration.
  • Data model acceleration is most efficient if the root event datasets and root search datasets being accelerated include in their initial constraint search the index(es) that Splunk software should search over. A single high-performance analytics store can span across several indexes in multiple indexers. If you know that all of the data that you want to pivot on resides in a particular index or set of indexes, you can speed things up by telling Splunk software where to look. Otherwise the Splunk software wastes time accelerating data that is not of use to you.

After you turn on acceleration for a data model

After you activate persistent acceleration for your data model, the Splunk software begins building a data model acceleration summary for the data model that spans the summary range that you've specified. Splunk software creates the .tsidx files for the summary in indexes that contain events that have the fields specified in the data model. It stores the .tsidx files parallel to their corresponding index buckets in a manner identical to that of report acceleration summaries.

After the Splunk software builds the data model acceleration summary, it runs scheduled searches on a 5 minute interval to keep it updated. Every 30 minutes, the Splunk software removes old, outdated .tsidx summary files. You can adjust these intervals in datamodels.conf and limits.conf, respectively.

A few facts about data model acceleration summaries:

  • Each bucket in each index in a Splunk deployment can have one or more data model acceleration summary .tsidx files, one for each accelerated data model for which it has relevant data. These summaries are created as data is collected
  • Summaries are restricted to a particular search head (or search head cluster GUID) to account for different extractions that may produce different results for the same search string, unless you have set up your Splunk platform deployment to share data model acceleration summaries among search heads or search head clusters. See Share data model acceleration summaries among search heads.
  • You can only accelerate data models that you have shared to all users of an app or shared globally to all users of your Splunk deployment. You cannot accelerate data models that are private. This prevents individual users from taking up disk space with private data model acceleration summaries.

Note: If necessary, you can configure the location of data model acceleration summaries for non SmartStore indexes via indexes.conf.

About the summary range

The Summary Range determines how far back in time the Splunk platform maintains summary data for a persistently accelerated data model. An empty value represents All Time. Summary Range has the same meaning for summaries associated with SmartStore and non-SmartStore indexes.

Data model acceleration summaries are created on a per-bucket basis, so their time coverage is approximate. A summary can contain some data older than its configured range when a source bucket spans the range boundary. After the initial summary build, scheduled summarization searches add newly eligible data and periodically remove summary data that falls outside a finite Summary Range.

Note: If you do not provide a Summary Range value, meaning the range is All Time, the Summary Range does not cause older summary data to age out.

The Splunk platform attempts to maintain summary data up to the configured Summary Range, subject to the availability and retention of the source data and any configured Backfill Range. Summary data is associated with source index buckets. When a source bucket freezes and is deleted or archived, the platform removes the corresponding summary information.

The effective coverage of a data model acceleration summary cannot exceed the retention of its source indexes. The source buckets of non-SmartStore indexes normally roll from cold to frozen, while source buckets of SmartStore indexes normally roll directly from warm to frozen. An All Time Summary Range does not override source-index retention or other summary-storage limits. See Configure data retention for SmartStore indexers in Manage Indexers and Indexer Clusters.

Searches generally complete fastest when their time range is fully covered by the available summary. If a search is only partially covered, the Splunk platform uses the summary for the covered portion and searches source _raw data for the remainder, provided that source data is still searchable.

Define a Summary Range that is broad enough to cover the searches you commonly run, while accounting for the additional storage and summarization workload created by a longer range. If building the initial summary for that range would be too resource-intensive, you can configure a shorter Backfill Range that causes Splunk software to build a partial summary that initially only covers this shorter time range, but which grows towards the Summary Range over time.

Note: Backfill Range is one of several settings related to Summary Range that you can use if you have a large Splunk deployment that involves multi-terrabyte datasets. This can lead to situations where the search required to build the initial data model acceleration summary runs too long and/or is resource intensive. For more information, see the subtopic Advanced configurations for persistently accelerated data models.

Summary range example

You create a data model and accelerate it with a Summary Range of 7 days. Assuming that summarization has completed successfully and the Backfill Range does not restrict coverage, Splunk software builds a summary for your data model that spans approximately the past 7 days. Splunk software then maintains it over time, periodically updating it with new data and removing data that is older than 7 days.

A search over the most recent 7 days uses the available data in the acceleration summary. However, if you run a similar search over the most recent 10 days, the search uses available summary data for approximately the most recent 7 days and runs over source data for the preceding 3 days. The portion of the search that runs over the summary is accelerated, but the portion of the search that runs over source data is not accelerated.

This assumes that the source indexes still retain all 10 days of data. If the source indexes retain fewer than 7 days the effective summary coverage can be shorter than the configured Summary Range.

Note: If your search uses the summariesonly argument to block the pulling of results from data that is not summarized, the search returns results from available summary data even when the search time range includes unsummarized data. This causes the search to complete faster with the trade off of getting results that might be incomplete. See Search data model acceleration summaries for more information.

How the Splunk platform builds data model acceleration summaries

When you activate acceleration for a data model, Splunk software builds the initial set of .tsidx file summaries for the data model and then runs scheduled searches in the background every 5 minutes to keep those summaries up to date. Each update ensures that the entire configured time range is covered without a significant gap in data. This method of summary building also ensures that late-arriving data is summarized without complication.

Parallel summarization

Data model acceleration summaries utilize parallel summarization by default. This means that by default, Splunk software can potentially run up to three concurrent search jobs to build and maintain .tsidx summary files instead of one. Parallel summarization is useful when the time that summarization searches take to complete is longer than the schedule on which new summarization searches are launched. Because parallel summarization allows summarization searches to run concurrently rather than sequentially, this feature can reduce the amount of time it might otherwise take to build and maintain data model summaries.

Note: There is a cost for this improvement in summarization search performance. The concurrent searches count against the total number of concurrent search jobs that your Splunk platform deployment can run. If you have several data models that require parallel summarization you may reach this concurrent search job limit.

Each data model summarization search works on a single bucket at a time before moving on to the next bucket. Summarization searches do not utilize thread level parallelization to summarize multiple buckets at once.

Manage parallel summarization with the Max Summarization Search Time, Maximum Concurrent Summarization Searches, and Summarization Period settings, which you can find under Advanced Settings when you manage data model acceleration in Splunk Web.

Prerequisites

Steps

  1. On your Splunk platform deployment, in Splunk Web, go to Settings > Data Models.
  2. Select Edit > Edit Acceleration for an accelerated data model that you want to edit.
  3. Open Advanced Settings.
  4. Review the Max Summarization Search Time, Maximum Concurrent Summarization Searches, and Summarization Period settings and adjust them as appropriate. See the following table for more information about these settings.
    Setting Description Default
    Max Summarization Search Time

    Set the maximum run time for searches that create or update the data model acceleration summary.

    Max Summarization Search Time is an approximate limit that causes a given summarization search to cease so that a new summarization search can begin. If a given data model acceleration search is in the middle of summarizing a bucket when it stops, the subsequent search will pick up summarizing that bucket where the first search left off.

    If you find that you are experiencing lags that are slowing down summarization and are making it harder to search on recent events, consider reducing the Max Summarization Search Time value. See When summary populating searches take too long to run.

    Note: Reducing the Max Summarization Search Time to a value below the average runtime of your data model acceleration searches will not cause your data model acceleration summary to be created faster.
    		1 hour
    Summarization Period Set the cron expression that the search scheduler uses to launch searches that create or update the data model acceleration summary on a regular interval, such as every five minutes or at the start of each hour. */5 * * * * (every five minutes)
    Maximum Concurrent Summarization Searches

    Set the maximum number of summarization searches that Splunk software can run concurrently. Searches might run concurrently when the time it takes for summarization searches to complete exceeds the Summarization Period for those searches.

    Be careful about setting Maximum Concurrent Summarization Searches higher than 3. If you have too many summarization searches running concurrently, there can be performance impacts to the system in terms of memory, CPU, and disk read/write operations.

    In addition, when a significant number of data models have Maximum Concurrent Summarization Searches set to 3 or higher, automatic summarization search quota limits might be reached, which might cause data model summarization searches to be skipped.

    		 3
  5. Select Save to save your changes.

Note: If your Splunk platform implementation does not use search head clustering and you find that the searches that build and maintain your data model acceleration summaries cause your implementation to reach or exceed concurrent search limits, consider lowering the Maximum Concurrent Summarization Searches setting.

Example of parallel summarization

Say there is an accelerated data model named DM1. Data model DM1 has a Summarization Period that causes the search scheduler to launch a new summary update search for DM1 every 5 minutes. DM1 has a Max Summarization Search Time of 600 seconds, or 10 minutes. For DM1, Maximum Concurrent Summarization Searches is set to 3.

Time: Now Time: Now + 5 minutes Time: Now + 10 minutes
Currently no summarization searches are running for data model DM1. Operating on DM1's Summarization Period, the search scheduler launches a summarization search for DM1. We'll call this search Summarization Search 1.

Five minutes later, Summarization Search 1 is still running because the Max Summarization Search Time for data model DM1 is 600 seconds, or 10 minutes.

The search scheduler checks to see if it can start another summarization search for data model DM1 even though one is currently in progress. The search scheduler finds that Maximum Concurrent Summarization Searches is set to 3, and only one search is currently running, so the scheduler launches a second search that we will name Summarization Search 2.

Ten minutes later, Summarization Search 1 has completed after running for 423 seconds, or just over 7 minutes.

Summarization Search 2 is still running because the Max Summarization Search Time for data model DM1 is 600 seconds, or 10 minutes.

The search scheduler checks to see if it can start another summarization search for data model DM1 even though one is currently in progress. The search scheduler finds that Maximum Concurrent Summarization Searches is set to 3, and only one search is currently running, so the scheduler launches Summarization Search 3.
Summarization searches running for data model DM1: Summarization Search 1 Summarization searches running for data model DM1: Summarization Search 1 and Summarization Search 2 Summarization searches running for data model DM1: Summarization Search 2 and Summarization Search 3

As you can see data model DM1 never quite reaches the Maximum Concurrent Summarization Searches limit, and it probably won't unless the summarization searches take much longer to complete and the administrator for DM1 raises the Max Summarization Search Time to accommodate them. However, if that is happening, the administrator might want to investigate whether there are ways to improve the scalability of data model DM1.

Review summary creation metrics

The speed of summary creation depends on the amount of events involved and the size of the summary range. You can track progress towards summary completion on the Data Models management page. Find the accelerated data model that you want to inspect, expand its row, and review the information that appears under ACCELERATION.

6.0 dm acceleration metrics.png

Status tells you whether the acceleration summary for the data model is complete. If it is in Building status it will tell you what percentage of the summary is complete. Data model acceleration summaries are constantly updating with new data. A summary that is "complete" now will return to "building" status later when it updates with new data.

When the Splunk software calculates the acceleration status for a data model, it bases its calculations on the Schedule Window that you have set for for the data model. However, if you have set a backfill relative time range for the data model, that time range is used to calculate acceleration status.

You might set up a backfill time range for a data model when the search that populates the data model acceleration summaries takes an especially long time to run. See Advanced configurations for persistently accelerated data models.

Verify that the Splunk platform is scheduling summary update searches

You can verify that Splunk software is scheduling searches to update your data model acceleration summaries. Inlog.cfg, set category.SavedSplunker=DEBUG and then watch scheduler.log for events like:

CODE 
04-24-2013 11:12:02.357 -0700 DEBUG SavedSplunker - Added 1 scheduled searches for accelerated datamodels to the end of ready-to-run list

When the data model definition changes and your summaries have not been updated to match it

When you change the definition of an accelerated data model, it takes time for Splunk software to update its summaries so that they reflect this change. In the meantime, when you run Pivot searches (or tstats searches) that use the data model, it does not use the summaries that are older than the new definition, by default. This ensures that the output you get from Pivot for the data model always reflects your current configuration.

If you know that the old data is "good enough" you can take advantage of an advanced performance feature that lets the data model return summary data that has not yet been updated to match the current definition of the data model, using a setting called allow_old_summaries, which is set to false by default.

  • On a search by search basis: When running tstats searches that select from an accelerated data model, set the argument allow_old_summaries=t.
  • For your entire Splunk deployment: Go to limits.conf and change the allow_old_summaries parameter to true.

Capacity planning summary size on disk for accelerated data models

How to use the data model Size on Disk metric on the Data Models management page to track the total size of a data model's summary on disk.

When planning capacity for accelerated data models in Splunk, it’s important to understand how summary data affects disk usage so you can make informed decisions about resource allocation and accurately monitor storage requirements. The following sections discuss how to interpret the Size on Disk metric for accelerated data models and highlight special considerations and limitations when working with models that have multiple root objects.

Data model acceleration summary size on disk

You can use the data model Size on Disk metric on the Data Models management page to track the total size of a data model's summary on disk. Because summaries can take up a significant amount of space, it's important to pay attention to the amount of physical disk space an accelerated data model needs, and avoid overusing data model acceleration. For example, you might want to reserve data model acceleration for data models whose pivots are heavily used in dashboard panels.

The amount of space that a data model takes up is related to the number of events that you are collecting for the summary range you have chosen. It can also be negatively affected if the data model includes fields with high cardinality (that have a large set of unique values), such as a Name field.

If you are particularly size constrained, you might want to test the amount of space a data model acceleration summary will take up by activating acceleration for a small Summary Range first, and then moving to a larger range if you think you can afford it.

Size on disk limitations for accelerated data models with multiple root objects

The Size on Disk metric on the Data Models management page displays the most up-to-date value for the amount of disk space used by an accelerated data model. However, if your accelerated data model contains multiple root objects, the Size on Disk value might be inaccurate because it accounts for only one root object. As a result, the value that is displayed for Size on Disk represents only partial disk usage, which can be significantly lower than the actual total size occupied by the entire data model on disk.

To get an accurate measurement of disk usage for accelerated data models with multiple root objects, use the data in the disk_objects.log file in the _introspection index instead of relying on the Size on Disk value in the Data Models management page. The disk_objects.log file, located at $SPLUNK_HOME/var/log/introspection/disk_objects.log, accurately reflects the total size used by the accelerated data model on each index. You can run the following search against the _introspection index to obtain this value for your capacity planning needs. This example search displays the latest disk space, in MB, that is occupied by a data model called DM_search_internal_audit_logs on each indexer.

CODE 
index=_introspection host=idx* sourcetype=”splunk_disk_objects” component=Summaries “data.name”=DM_search_internal_audit_logs |dedup host |table data.total_size

Your results look like this:

host data.total_size
idx-i-0b59de6ad026398a8.dma-test.stg.splunkcloud.com 5.086
idx-i-066241aec65c3ce7c.dma-test.stg.splunkcloud.com 5.105
idx-i-085987211d56467ea.dma-test.stg.splunkcloud.com 4.85

For information about the disk_objects log, see disk_objects.log.

Where the Splunk platform creates and stores data model acceleration summaries

Persistent data model acceleration summaries consist of .tsidx files associated with source index buckets that contain the summarized events. A single accelerated data model can span multiple indexes and indexers. If those source indexes include both SmartStore and non-SmartStore indexes, each part of the summary follows the storage behavior of its source index.

Note: This section describes persistent data model acceleration. Ad hoc data model acceleration summaries are created in dispatch directories on the search head and are not stored with source index buckets. See About ad hoc data model acceleration.

Non-SmartStore indexes

For a non-SmartStore index, the Splunk platform creates summary files on an indexer, parallel to the corresponding hot, warm, or cold source bucket. When the source bucket freezes and the platform deletes or archives it, the platform also removes the summary data associated with that bucket.

By default, these summaries use the predefined _splunk_summaries volume and the following tstatsHomePath value:

CODE 
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary

The resulting local path has this general structure:

CODE 
$SPLUNK_DB/<index_name>/datamodel_summary/<bucket_id>/<search_head_or_pool_id>/DM_<datamodel_app>_<datamodel_name>

The _splunk_summaries volume has no independent size limit by default. This does not mean that summary data is necessarily retained indefinitely. The Summary Range and source-bucket lifecycle continue to apply.

For non-SmartStore indexes, you can:

  • Override tstatsHomePath with another local volume and path.

  • Configure different tstatsHomePath values for individual indexes.

  • Set maxVolumeDataSizeMB on a local summary volume to configure size-based retention.

CAUTION: tstatsHomePath specifies storage for persistent data model acceleration summaries. Do not confuse it with tsidxStatsHomePath, which specifies storage for namespace data created by the tscollect command.

SmartStore indexes

For a SmartStore index, an indexer creates summary data associated with the source index buckets. In an indexer cluster, the peer that is primary for an associated bucket creates the summary. The peer then uploads the summary to remote storage.

When a peer needs the summary for a search, its Cache Manager fetches the required files into the peer's local cache. The remote summary copy remains available to the other peers. A locally cached summary copy is temporary and can be evicted when Cache Manager needs to free local space. Eviction of a local summary copy does not remove the summary from remote storage. See How indexing works in SmartStore and The SmartStore cache manager in Manage Indexers and Indexer Clusters.

Do not add or override tstatsHomePath for a SmartStore index. In a deployment containing both SmartStore and non-SmartStore indexes, avoid placing a customized tstatsHomePath in the [default] stanza, because SmartStore indexes might inherit it. Scope customized paths to the applicable non-SmartStore index stanzas.

How clusters handle data model acceleration summaries

For non-SmartStore indexes, indexer clusters do not replicate data model acceleration summaries by default. In general, only primary bucket copies have associated summaries. If primacy moves to another bucket copy, the summary can be unavailable until a subsequent summarization search detects that it is missing and regenerates it.

Note: For non-SmartStore indexes, when summary replication is turned off, summarization searches run on primary bucket copies by default. The Splunk platform does not automatically copy summaries to the other searchable bucket copies. If primacy moves, a summary might remain on the previous primary, but the new primary might not have a usable copy until it is regenerated by a subsequent summarization search.

You can configure summary replication for non-SmartStore indexes so that all searchable bucket copies have associated summaries. See How indexer clusters handle report and data model acceleration summaries, in the Managing Indexers and Clusters of Indexers manual.

For SmartStore indexes, summary replication is unnecessary and unsupported. The summary in remote storage can be fetched through Cache Manager by an eligible peer that is handling the associated source bucket.

Manage retention for data model acceleration summaries

Data model acceleration summary retention is governed by:

  • The data model's Summary Range.

  • The retention of the associated source index buckets.

  • For non-SmartStore indexes only, an optional local-volume size limit.

The summarization process aims to periodically remove summary data older than a finite Summary Range. When a source index bucket freezes, the platform also removes the summary data associated with that bucket.

Note: Summary Range retention is approximate. The platform removes a summary when its source bucket's end time is older than the Summary Range cutoff. A retained boundary bucket can therefore include summarized events older than the configured range.

Additional retention controls apply to data model acceleration summaries that have SmartStore source indexes.

Configure size-based retention for non-SmartStore indexes

For non-SmartStore indexes, data model acceleration summaries use the local _splunk_summaries volume by default. The volume has no size limit by default.

To configure an additional storage limit:

  1. Identify or create the local volume that will store the summaries.

    You can use _splunk_summaries, another volume on the filesystem that contains the index buckets, or a volume on a separate filesystem. If a volume also contains other index data, maxVolumeDataSizeMB applies to all data that references the volume. Use a dedicated volume if you need an independent limit for data model acceleration summaries.

  2. Set maxVolumeDataSizeMB for the volume.

    When a dedicated data model acceleration summary volume exceeds this limit, the volume manager removes the summaries associated with the oldest eligible source buckets and leaves a done marker in each trimmed summary directory so that the corresponding summary is not automatically rebuilt.

    If the volume also contains other index data or summary types, the limit applies across all eligible contents of that volume, in which case trimming is not restricted to data model acceleration summaries.

  3. If you created another volume, set tstatsHomePath for each applicable non-SmartStore index so that it references that volume.

For example:

CODE 
[volume:local_dma_summaries]
path = /mnt/dma_summaries
maxVolumeDataSizeMB = 100000

[main]
tstatsHomePath = volume:local_dma_summaries/main/datamodel_summary

If all indexes in your deployment are non-SmartStore, you can instead add maxVolumeDataSizeMB to the predefined _splunk_summaries volume and continue using the default tstatsHomePath.

In a mixed deployment, use a local summary volume referenced exclusively by non-SmartStore indexes. The Splunk platform ignores maxVolumeDataSizeMB for remote volumes and local volumes that contain remote-storage-enabled indexes. Do not put a customized tstatsHomePath in [default] when SmartStore indexes might inherit it.

Restart the affected indexers after adding a volume or after changing tstatsHomePath or maxVolumeDataSizeMB.

See Configure maximum index size in Manage Indexers and Indexer Clusters.

Manage retention for SmartStore indexes

Do not set or override tstatsHomePath for a SmartStore index. SmartStore ignores maxVolumeDataSizeMB, so the non-SmartStore volume procedure cannot impose a separate size limit on SmartStore data model acceleration summaries.

Use these mechanisms to manage retention for data model acceleration summaries that are associated with SmartStore indexes:

  • Configure a finite Summary Range. The summarization process removes data that falls outside the range. Reducing the Summary Range is the most direct way to reduce how much historical data the acceleration summary covers.

    Note: Summary Range retention is approximate and operates on bucket-associated summaries. The platform removes a summary when its source bucket's end time is older than the Summary Range cutoff. A retained boundary bucket can therefore include summarized events older than the configured range.

  • Freeze the associated source SmartStore index buckets. If your environment uses the Victoria Experience, SmartStore freezes index buckets when maxGlobalRawDataSizeMB or frozenTimePeriodInSecs are reached. If your environment uses the Classic Experience, SmartStore freezes index buckets when maxGlobalDataSizeMB, maxGlobalRawDataSizeMB, or frozenTimePeriodInSecs is reached. In either case, the first applicable limit reached can cause the qualifying source buckets to freeze and be removed from the platform.

    When a source bucket is removed, the platform also removes its associated data model acceleration summary data.

    Data model acceleration summary files are excluded from the calculations for the SmartStore size limits. These settings can therefore remove summary data indirectly by freezing source buckets, but they do not impose a size limit on the summaries themselves. See Configure data retention for SmartStore indexes in Manage Indexers and Indexer Clusters.

Note: Cache eviction is not summary removal. Cache Manager can evict a locally cached copy of a summary to free local disk space. The authoritative copy remains in remote storage and can be fetched again when needed. See the information about data model acceleration summaries in How indexing works in SmartStore in Manage Indexers and Indexer Clusters.

Search data model acceleration summaries

You can search the summary for a specific accelerated data model in Search with the tstats command.

tstats can sort through the full set of .tsidx file summaries that belong to your accelerated data model even when they are distributed among multiple indexes.

This can be a way to quickly run a stats-based search against a particular data model just to see if it's capturing the data you expect for the summary range you've selected.

To do this, you identify the data model using FROM datamodel=<datamodel-name>:

CODE 
| tstats avg(foo) FROM datamodel=buttercup_games WHERE bar=value2 baz>5

This search returns the average of the field foo in the "Buttercup Games" data model acceleration summaries, specifically where bar is value2 and the value of baz is greater than 5.

Note: You don't have to specify the app of the data model as Splunk software takes this from the search context (the app you are in). However you cannot search an accelerated data model in App B from App A unless the data model in App B is shared globally.

Using the summariesonly argument

When you run a tstats search on an accelerated data model where the search has a time range that extends past the summarization time range of the data model, the search will generate results from the summarized data within that time range and from the unsummarized data that falls outside of that time range. This means that the search will be only partially accelerated. It can quickly pull results from the data model acceleration summary, but it slows down when it has to pull results from the "raw" unsummarized data outside of the summary.

If you do not want your tstats search to spend time pulling results from unsummarized data, use the summariesonly argument. This tstats argument ensures that the search generates results only from the TSIDX data in the data model acceleration summary. Non-summarized results are not provided.

For more about the tstats command, see the entry for tstats in the Search Reference.

Turn on multi-eval to improve data model acceleration

Searches against root-event datasets within data models iterate through many eval commands, which can be an expensive operation to complete during data model acceleration. You can improve the data model search efficiency by enabling multi-eval calculations for search in limits.conf.

CODE 
enable_datamodel_meval = <bool>
* Allow concatenation of successively occurring evals into a single comma-separated eval during the generation of data model searches.
* default: true

If you turned off automatic rebuilds for any accelerated data model, you will need to rebuild that data model manually after enabling multi-eval calculations. For more information about rebuilding data models, see Manage data models.

Advanced configurations for persistently accelerated data models

There are a few situations that may require you to set up advanced configurations for your persistently accelerated data models.

When summary-populating searches take too long to run

If your Splunk deployment processes an extremely large amount of data on a regular basis you may find that the initial creation of persistent data model acceleration summaries is resource intensive. The searches that build these summaries may run too long, causing them to fail to summarize incoming events. To deal with this situation, Splunk software gives you two configuration parameters that you can set through Splunk Web. These parameters are Max Summarization Search Time and Backfill Range.

You can find both of these settings by opening the Edit Acceleration window for a data model and selecting Advanced Settings.

Change the maximum period of time that a summary-populating search can run

Max Summarization Search Time causes summary populating searches to quit after a specified amount of time has passed. After a summary-populating search stops, Splunk software runs a search to catch all of the events that have come in since the initial summary-populating search began, and then it continues updating the summary where the last summary-populating search left off.

For example, let's say you have activated acceleration for a data model, and you want its summary to retain events for the past three months. Because your organization indexes large amounts of data, the search that initially creates this summary should take about four hours to complete. Unfortunately you can't let the search run uninterrupted for that amount of time because it might fail to index some of the new events that come in while that four-hour search is in process.

Max Summarization Search Time stops the first search after an hour, and a second search takes its place. This second search pulls in the new events that have come in over the past hour. It then continues running to add more events from the last three months to the summary. This second search also stops after an hour and the process repeats until the summary of the last three months of data is complete.

If you find that your summarization searches have data collection lags that make it difficult to search on recent events, adjust your summarization settings to remove those lags. Continuing with our example, let's say your data model has the default acceleration settings:

Setting Value
Summarization Period Every 5 minutes
Maximum Concurrent Summarization Searches 3 searches
Max Summarization Search Time 3600 seconds (1 hour) - default setting

With these settings, if the data model summarization searches each run as long or longer than the default Max Summarization Search Time of one hour, that data model would likely experience periodic lags in data collection.

The following diagram illustrates that there will be a lag of 45 minutes between the start of search 3 and the start of search 4. This is due to two facts: Only 3 searches can run concurrently and each search takes about an hour to complete. Search 4 must collect the new data that comes in during that 45 minute lag, and search 4 can't add this new data to the summary index until it completes.

This diagram shows timelines for the first four summarization searches for a data model that has the default acceleration settings. The first search runs for an hour. The second search starts 5 minutes after the first search and also runs for an hour. The third search starts five minutes after the second search and also runs for an hour. The fourth search, however, must wait for the first search to complete because the Maximum Concurrent Summarization Searches setting is set to 3 searches. This means that Search 4 starts 45 minutes after the third search begins, and its results won't be added to the summary until it completes.

To reduce this lag, you must change the settings. You might increase Maximum Concurrent Summarization Searches, or reduce the Summarization Period, or reduce the Max Summarization Search Time.

Here is what might happen if you reduce the Max Summarization Search Time to 1200 seconds (20 minutes). The data lag is removed, making it easier to search recent data.

This diagram shows timelines for the first four summarization searches for a data model that has the default acceleration settings. The first search runs for 20 minutes. The second search starts 5 minutes after the first search and also runs for 20 minutes. The third search starts five minutes after the second search and also runs for 20 minutes. The fourth search must wait for the first search to complete, but this means that it starts only 10 minutes after the third search, and will only take 10 more minutes to complete.

Note that when you reduce the data collection lag by lowering the Max Summarization Search Time to 1200, you do not have much capacity to increase Maximum Concurrent Summarization Searches value above 3.

For more information about Max Summarization Search Time see Parallel summarization.

Set a backfill time range that is shorter than the summary time range

If you are indexing a tremendous amount of data with your Splunk deployment and you don't want to adjust the Max Summarization Search Time range for a slow-running summary-populating search, you have an alternative option: the Backfill Range setting.

Backfill Range creates a second "backfill time range" that you set within the summary range. Splunk software builds a partial summary that initially only covers this shorter time range. After that, the summary expands with each new event summarized until it reaches the limit of the larger summary time range. At that point the full summary is complete and events that age out of the summary range are no longer retained.

For example, say you want to set your Summary Range to 1 Month but you know that your system would be taxed by a search that built a summary for that time range. To deal with this, you set the Backfill Range to -7d to run a search that creates a partial summary that initially just covers the past week. After that limit is reached, Splunk software only adds new events to the summary, causing the range of time covered by the summary to expand. But the full summary still retains events only for one month. Once the partial summary expands to the full Summary Range of the past month, it starts dropping its oldest events, just like an ordinary data model acceleration summary does.

When you do not want persistently accelerated data models to be rebuilt automatically

By default Splunk software automatically rebuilds persistently accelerated data models whenever it finds that those models are outdated. Data models can become outdated when the current data model search does not match the version of the data model search that was stored when the data model was created.

This can happen if the JSON file for an accelerated model is edited on disk without first disabling the model's acceleration. It can also happen when changes are made to knowledge objects that are interdependent with the data model search. For example, if the data model constraint search references an event type, and the definition of that event type changes, the constraint search will return different results than it did before the change. When the Splunk software detects this change, it rebuilds the data model.

CAUTION: Accelerated data models that have Automatic Rebuilds turned on might experience unbounded memory growth due to extensive search expansions, resulting in Out of Memory errors. In addition, the work of rebuilding large data model acceleration summaries each time a referenced knowledge object changes might be a burden on the system resources of your Splunk platform deployment. Turn off Automatic Rebuilds for accelerated data models that have large summaries on disk.

To turn off automatic rebuilds for a specific persistently accelerated data model, open the Edit Acceleration window for the data model and deselect Automatic Rebuilds.

When Automatic Rebuilds is turned off for an accelerated data model, you must manually initiate rebuilds of that data model. You can manually rebuild an accelerated data model through the Data Model Manager page, by expanding the row for the affected data model and selecting Rebuild.

See Manage data models.

You can search an out-of-date data model acceleration summary by adding allow_old_summaries = true to a tstats search of that summary. For more information about the allow_old_summaries argument, see the tstats reference topic in the Search Reference.

About ad hoc data model acceleration

Even when you're building a pivot that is based on a data model dataset that is not accelerated in a persistent fashion, that pivot can benefit from what we call "ad hoc" data model acceleration. In these cases, Splunk software builds a summary in a search head dispatch directory when you work with a dataset to build a pivot in the Pivot Editor.

The search head begins building the ad-hoc data model acceleration summary after you select a dataset and enter the pivot editor. You can follow the progress of the ad hoc summary construction with the progress bar:

6.0 pivot progressbar.png

When the progress bar reads Complete, the ad hoc summary is built, and the search head uses it to return pivot results faster going forward. But this summary only lasts while you work with the dataset in the Pivot Editor. If you leave the editor and return, or switch to another dataset and then return to the first one, the search head will need to rebuild the ad hoc summary.

Ad hoc data model acceleration summaries complete faster when they collect data for a shorter range of time. You can change this range for root datasets and their children by resetting the time Filter in the Pivot Editor. See "About ad hoc data model acceleration summary time ranges," below, for more information.

Ad hoc data model acceleration works for all dataset types, including root search datasets that include transforming commands and root transaction datasets. Its main disadvantage against persistent data model acceleration is that with persistent data model acceleration, the summary is always there, keeping pivot performance speedy, until acceleration is deactivated for the data model. With ad hoc data model acceleration, you have to wait for the summary to be rebuilt each time you enter the Pivot Editor.

About ad hoc data model acceleration summary time ranges

The search head always tries to make ad hoc data model acceleration summaries fit the range set by the time Filter in the Pivot Editor. When you first enter the Pivot Editor for a dataset, the pivot time range is set to All Time. If your dataset represents a large dataset this can mean that the initial pivot will complete slowly as it builds the ad hoc summary behind the scenes.

When you give the pivot a time range other than All Time, the search head builds an ad hoc summary that fits that range as efficiently as possible. For any given data model dataset, the search head completes an ad hoc summary for a pivot with a short time range quicker than it completes when that same pivot has a longer time range.

The search head only rebuilds the ad hoc summary from start to finish if you replace the current time range with a new time range that has a different "latest" time. This is because the search head builds each ad hoc summary backwards, from its latest time to its earliest time. If you keep the latest time the same but change the earliest time the search head at most will work to collect any extra data that is required.

Note: Root search datasets and their child datasets are a special case here as they do not have time range filters in Pivot (they do not extract _time as a field). Pivots based on these datasets always build summaries for all of the events returned by the search. However, you can design the root search dataset's search string so it includes "earliest" and "latest" dates, which restricts the dataset represented by the root search dataset and its children.

How ad hoc data model acceleration differs from persistent data model acceleration

Here's a summary of the ways in which ad hoc data model acceleration differs from persistent data model acceleration:

  • Ad hoc data model acceleration takes place on the search head rather than the indexer. Locating the data model acceleration process on the search head allows it to accelerate all three dataset types (event, search, and transaction).
  • Splunk software creates ad hoc data model acceleration summaries in dispatch directories at the search head. It creates and stores persistent data model acceleration summaries in your indexes alongside index buckets.
  • Splunk software deletes ad hoc data model acceleration summaries when you leave the Pivot Editor or change the dataset you are working on while you are in the Pivot Editor. When you return to the Pivot Editor for the same dataset, the search head must rebuild the ad hoc summary. You cannot preserve ad hoc data model acceleration summaries for later use.
    • Pivot job IDs are retained in the pivot URL, so if you quickly use the back button after leaving Pivot (or return to the pivot job with a permalink) you may be able to use the ad-hoc summary for that job without waiting for a rebuild. The search head deletes ad hoc data model acceleration summaries from the dispatch directory a few minutes after you leave Pivot or switch to a different model within Pivot.
  • Ad hoc acceleration does not apply to reports or dashboard panels that are based on pivots. If you want pivot-based reports and dashboard panels to benefit from data model acceleration, base them on datasets from persistently accelerated event dataset hierarchies.
  • Ad hoc data model acceleration can potentially create more load on your search head than persistent data model acceleration creates on your indexers. This is because the search head creates a separate ad hoc data model acceleration summary for each user that accesses a specific data model dataset in Pivot that is not persistently accelerated. On the other hand, summaries for persistently accelerated data model datasets are shared by each user of the associated data model. This data model acceleration summary reuse results in less work for your indexers.