Investigate skipped scheduled searches

The CMC Skipped Scheduled Searches dashboard provides information to Splunk Cloud Platform administrators on skipped searches and search errors. Use this dashboard to identify why the Splunk platform can't process your scheduled searches and take steps to correct the issues.

See also Prioritize concurrently scheduled reports in Splunk Web and Offset scheduled search start times in the Splunk Cloud Platform Reporting Manual. 

Review the Skipped Scheduled Searches dashboard

This dashboard includes six panels of summary, graphical, and tabular data. Filter the results by specifying a time range and opting to include or exclude acceleration searches.

To investigate your panels, go to Cloud Monitoring Console > Search > Skipped Scheduled Searches. Use the following table to understand the dashboard interface.

Panel or Filter Description
Time Range Set the time range for the data display.
Include Acceleration Searches Acceleration searches are summaries of large datasets, used to help efficiently report on large volumes of data.

Select Yes to include summary-based acceleration searches in the displayed results. Select No to include only searches run on the complete dataset in the results.

Total Skipped Searches Shows the total number of skipped searches.
Scheduled Search Skip Ratio Shows the percentage of your scheduled searches that had to be skipped.
Skipped Scheduled Searches Detail Shows a table with Group by filter.
Skipped Searches Shows a bar graph of skipped scheduled searches with Group by filter.
Skipped Scheduled Searches by Name and Reason Shows a table that lists the following:
  • Report name
  • Skip reason and count
  • Alert actions
  • Total skips
Scheduler Errors and Warnings Shows a table that lists the following:
  • Error message
  • Count, or number of times the message was issued
  • Percent of Total, indicating what percent of the total this error was issued

Interpret skipped scheduled searches results

If you are skipping searches, it can be indicative of the following possible problems with your search scheduling or query formation:

  • You have scheduled too many searches to run at the same time. Alleviate this problem by staggering the scheduled searches.
  • You have a search that attempts to run before the previously scheduled search has completed. For example, you schedule Search_A to run every 5 minutes, but the first instance of the search takes 10 minutes to complete. The next time the search is scheduled to run, it is skipped because the first search has not yet completed. Correct this issue by either adjusting the time range (set it to 10 minutes instead of 5) or optimizing your search to improve its performance. See About search optimization in the Splunk Cloud Platform Search Manual.
  • You have skipped searches because your users have met the threshold for concurrency limits that you set in your Splunk System Limits. This is expected behavior, but it may also indicate that your users need help in optimizing their searches.

To check for skipped searches, perform these steps:

  1. In the Time Range field of Skipped Scheduled Searches, select 24 hours to get a better picture of your searches historically.
  2. In the Skipped Scheduled Searches Detail panel, sort by Reason. Frequently, there are a number of skipped searches for the same reason. Note the primary reason or reasons that searches are skipped.
  3. Scroll down to see which report is generating the primary issues. Note the report name and determine the following:
    • If this is an expected behavior, you don't need to research any further.
    • If the skipped searches are unexpected, continue to the next step.
  4. Go to Settings > Searches Reports and Alerts.
  5. If you know the App associated with the search or report, you can sort by the App. Otherwise, search by the report or search name.
  6. Locate the needed search or report and select it to open the Edit Search dialog box.
  7. Determine if the problem is with the search formation or the scheduling:
    • If you need to troubleshoot the formation of the search, look for wild cards and check whether an index is specified.
    • If scheduling is the problem, continue to the next step.
  8. Go to Edit > Edit Schedule to review the schedule for the search.
  9. Verify that the schedule for the report or search is in line with how long the search takes to complete. For example, if the report runs every hour but it takes 1.5 hours to run the search, the searches are skipped.

Enable the Skipped Scheduled Searches alert

Skipped searches can be indicators of non-optimal performance in your deployment. A high ratio of skipped scheduled searches can indicate the following:

  • The number of searches being run exceeds your deployment's capacity.
  • The searches being run are taking too long, or using large amounts of memory or CPU.

CMC provides an alert that lets you know when the ratio of skipped scheduled searches exceeds 20% in a 60-minute period. For more information about using this alert, see Use the Alerts panel. For general information about managing alerts, see the Splunk Cloud Platform Alerting Manual.