Overview of summary-based search acceleration
Searches over large datasets can take a long time to complete. This isn't a problem if you run such searches on an infrequent basis. But if you are like many users of Splunk Cloud Platform, you do not have this luxury. Large dataset searches must be run on schedules, made the basis for panels in popular dashboards, or run ad-hoc frequently by large numbers of users.
Splunk Cloud Platform offers several approaches to speeding up searches of large datasets. One of these approaches is summary-based search acceleration. This is where you create a data summary that is populated by background runs of a slow-completing search. The summary is a smaller dataset that contains only data that is relevant to your search. When you run the search against the summary, the search should complete much faster.
There are three methods of summary-based search acceleration:
- Report acceleration - Uses automatically-created summaries to speed up completion times for certain kinds of event searches.
- Data model acceleration - Uses automatically-created event summaries to speed up completion times for data-model-based searches.
- Summary indexing - Populates a summary index using a scheduled search that you define. You can create summary indexes of event data, or you can convert your event data into metrics and summarize it in metrics summary indexes.
Comparing summary-based search acceleration methods
| Acceleration method | Description | Location of summary | When should you use it? | For more information |
|---|---|---|---|---|
| Report acceleration | Accelerates qualifying transforming searches of event data that have been saved as reports. Features automatic backfill for data interruptions. Similar saved searches can use the same acceleration summary when they are accelerated. | In .tsidx files, stored alongside buckets in indexers.
|
Use for any qualifying saved search that has 100k or more hot bucket events. Not all searches qualify. | Manage report acceleration |
| Data model acceleration | Accelerates searches run against qualifying data models by running those searches on a summary of the data model rather than the data model itself. Allows you to speed up searches against large and varied datasets. | In .tsidx files, stored alongside buckets in indexers.
|
Consider enabling acceleration for any qualifying data model. Data model acceleration can be faster than report acceleration, especially for relatively complicated searches. | Accelerate data models |
| Event summary indexing | Speeds up slow-completing transforming searches of event data by summarizing the events returned by the search in a separate events index. | In a summary index composed of summarized event data. You must predefine the event summary index if one does not already exist. | Create an event summary index if you want to speed up a transforming search that does not qualify for report acceleration. You might also want to create a summary index to keep certain data in an index with different data retention policies than your other indexes. | Use summary indexing for increased search efficiency |
| Metrics summary indexing | Speeds up slow-completing transforming searches of event data by converting the events returned by the search into metric data points and summarizing those metric data points in a separate metrics index. | In a summary index composed of aggregated metric data points. You must predefine the metric summary index if one does not already exist. | Use metrics summary indexing over event summary indexing if it makes sense to convert your event data into metrics data. Metrics summary indexes can provide faster search performance and more efficient data storage than event summary indexes. | Use summary indexing for increased search efficiency |
Batch mode search
Batch mode search is a feature that improves the performance and reliability of transforming searches. For transforming searches that don't require the events to be time-ordered, running in batch mode means that the search executes bucket-by-bucket (in batches), rather than over time. In certain reporting cases, this means that the transforming search can complete faster. Additionally, batch mode search improves the reliability for long-running distributed searches, which can fail when an indexer goes down while the search is running. In this case, Splunk software attempts to reconnect to the missing peer and retry the search.
Transforming searches that meet the criteria for batch mode search include:
- Generating and transforming searches (stats, chart, etc.) that do not include the
localizeortransactioncommands in the search. - Searches that are not real-time and not summarizing searches.
- Non-distributed searches that are not stateful streaming. (A streamstats search is an example of a stateful streaming search.)
Batch mode search is invoked from the configuration file, in the [search] stanza of limits.conf. Use the search inspector to determine whether or not a transforming search is running in batch mode.