Manage summary index gaps
The accuracy of your summary index searches can be compromised if the summary indexes involved have gaps in their collected data.
Gaps in summary index data can occur for a number of reasons:
- A summary index initially only contains events from the point that you start data collection: Don't lose sight of the fact that summary indexes won't have data from before the summary index collection start date--unless you arrange to put it in there yourself with the backfill script.
- Splunk deployment outages: If your Splunk deployment goes down for a significant amount of time, there's a good chance you'll get gaps in your summary index data, depending on when the searches that populate the index are scheduled to run.
- Searches that run longer than their scheduled intervals: If the search you're using to populate the summary index runs longer than the interval that you've scheduled it to run on, then you're likely to end up with gaps because Splunk software won't run a scheduled search again when a preceding search is still running. For example, if you were to schedule the index-populating search to run every five minutes, you'll have a gap in the index data collection if the search ever takes more than five minutes to run.
For general information about creating and maintaining summary indexes, see Use summary indexing for increased reporting efficiency.
Use the backfill script to add other data or fill summary index gaps
fill_summary_index.py script can only be run by Splunk Support.
If you have Splunk Enterprise, you can use the fill_summary_index.py script, which backfills gaps in summary index collection by running the saved searches that populate the summary index as they would have been executed at their regularly scheduled times for a given time range. In other words, even though your new summary index only started collecting data at the start of this week, if necessary, you can use fill_summary_index.py to fill the summary index with data from the past month.
In addition, when you run fill_summary_index.py you can specify an App and schedule backfill actions for a list of summary index searches associated with that App, or simply choose to backfill all saved searches associated with the App.
When you enter the fill_summary_index.py commands through the CLI, you must provide the backfill time range by indicating an "earliest time" and "latest time" for the backfill operation. You can indicate the precise times either by using relative time identifiers (such as -3d@d for "3 days ago at midnight") or by using UTC epoch numbers. The script automatically computes the times during this range when the summary index search would have been run.
The fill_summary_index.py script requires that you provide necessary authentication (username and password). If you know the valid Splunk Enterprise key when you invoke the script, you can pass it in via the -sk argument.
The script is designed to prompt you for any required information that you fail to provide in the command line, including the names of the summary index searches, the authentication information, and the time range.
Examples of fill_summary_index.py invocation
If this is your situation:
You need to backfill all of the summary index searches for the splunkdotcom App for the past month--but you also need to skip any searches that already have data in the summary index:
Then enter this into the CLI:
./splunk cmd python fill_summary_index.py -app splunkdotcom -name "*" -et -mon@mon -lt @mon -auth admin:changeme
If this is your situation:
You need to backfill the my_daily_search summary index search for the past year, running no more than 8 concurrent searches at any given time (to reduce impact on performance while the system collects the backfill data). The my_daily_search summary index search is owned by the "admin" role.
Then enter this into the CLI:
./splunk cmd python fill_summary_index.py -app search -name my_daily_search -et -y -lt now -j 8 -owner admin -auth admin:changeme
-owner argument for searches that are owned by a specific user or role.
What to do if fill_summary_index.py is interrupted while running
If fill_summary_index.py is interrupted, look for a log directory in the app that you are invoking the process from, such as Search. In that directory you should find an empty temp file named fsidx*lock.
Delete this temp file and restart fill_summary_index.py.
fill_summary_index.py arguments
To use the fill_summary_index.py script, enter the following command in the CLI:
python fill_summary_index.py
Add the required and optional fields to the command line from the following tables.
1, t, true, or yes for "true" and 0, f, false, or no for "false."
| Field | Value |
|---|---|
-et <string> |
Earliest time (required). Either a UTC time or a relative time string. |
-lt <string> |
Latest time (required). Either a UTC time or a relative time string. |
-app <string> |
The app context to use (defaults to None). |
-name <string> |
Specify a single saved search name. Can specify multiple times to provide multiple names. Use the wildcard symbol ("*") to specify all enabled, scheduled saved searches that have a summary index action. |
-names <string> |
Specify a comma seperated list of saved search names. |
-namefile <filename> |
Specify a file with a list of saved search names, one per line. Lines beginning with a # are considered comments and ignored. |
-owner <string> |
The user context to use (defaults to "None"). |
-index <string> |
Identifies the summary index that the saved search populates. If the index is not provided, the backfill script tries to determine it automatically. If this attempt at auto index detection fails, the index defaults to "summary". |
-auth <string> |
The authentication string expects either <username> or <username>:<password>. If only a username is provided, the script requests the password interactively. |
-sleep <float> |
Number of seconds to sleep between each search. Default is 5 seconds. |
-j <int> |
Maximum number of concurrent searches to run (default is 1). |
-dedup <boolean> |
When set to true, which is the default, this argument instructs the script to skip running the saved search for any scheduled timespans where data already exists in the summary index. The script uses a dedup search, either the default or a custom dedup search specified by -dedupsearch, to identify these timespans. Deduping the search prevents duplication of summary index data.
Note: To ensure that the
fill_summary_index.py script only executes summary index searches at times that correspond to missing data, do not use -dedup false when you run the script.
Note: The -dedup option is unrelated to the dedup command in SPL. Unlike the SPL command, the fill_summary_index.py script doesn't have the ability to perform event-level data analysis or determine whether certain events are duplicates of others.
|
-showprogress <boolean> |
When this argument is set to true, the script periodically shows the done progress for each currently running search that it spawns. If this argument is not used, its default is false. |
The following advanced arguments should not be used in most cases.
| Field | Value |
|---|---|
-trigger <boolean> |
When this argument is set to false, the script runs each search but does not trigger the summary indexing action. If this argument is not used, its default is true. |
-dedupsearch <string> |
Specifies a custom dedup search for use with the -dedup argument. The script uses the dedup search to find scheduled timespans that already have data in the summary index, so the data for those timespans can be skipped during backfill. If this argument is not specified, the script uses its default dedup search. Use -dedupsearch to override the default dedup search with your own custom dedup search. |
-namefield <string> |
Indicates the field in the summary index data that contains the name of the saved search that generated that data. |
-timefield <string> |
Indicates the field in the summary index data that contains the scheduled time of the saved search that generated that data. |