Rerun detections in Splunk Enterprise Security

Identify and rerun high priority detections in Splunk Enterprise Security when detections fail to run due to planned or unplanned outages such as rolling upgrades, restarts, resource contention, and so on.

Detections must be run continuously and on schedule to ensure that a security operations center (SOC) has complete coverage. However, when detections are not run during their scheduled time windows due to outages, threats can remain undetected. Manually adding findings to the analyst queue, hard coding time ranges to run detections, or rerunning detections one by one can overlook risk and is a complex, error-prone, and time intensive process.

Splunk Enterprise Security version 8.8 and higher provides insight into the detections that were missed during an outage and the duration for which a detection was not run. Based on these insights, you can rerun the detections to cover the outage period and ensure that findings are displayed in the analyst queue. This ensures that findings that indicate risk are not lost but are displayed in the analyst queue with clear indicators that they were created when the detection was rerun.

Splunk Enterprise Security can help do the following tasks:

  • Automatically surface every detection that was skipped, failed, or returned partial results during an outage​.

  • Select specific skipped detections or all skipped detections, and specific time windows to rerun the detections against historical data such as trigger conditions, risk scoring, and alert actions. This ensures that the detections are rerun with the same conditions as if they were run on schedule.

  • Surface findings from detections that are rerun in the analyst queue with their full context so that you can get a clear understanding of the evidence and risk.

Note: Splunk Enterprise Security retains 90 days of history for all detections that are run.
Splunk Enterprise Security automatically tracks the status of detection runs every hour across the entire detection library. Skipped detections are sorted by health score so that running the most important detections is prioritized. A visual heatmap coverage shows the time windows that were impacted by the outage and the reason for the outage.

Rerun skipped detections and review missed findings

Follow these steps to rerun skipped detections:

  1. In Splunk Enterprise Security, go to Security content and select Detection studio.
  2. Select Launchpad and then select Detection coverage to view the security coverage provided by the activated detections.

  3. Go to the Detection library that lists all the detections in your SOC.

  4. Go to Performance in the left panel and then turn on the toggle to specify how the detections are displayed. For example:

    • Skipped: Turn on to display all the detections that were skipped due to outages.

    • Failed: Turn on to display all the detections that failed to run due to syntax errors, SPL errors, and so on.

    • Partial: Turn on to display detections that started to run but the run was not completed.

  5. Go to the key indicator tiles to view the following:

    • Backfill activity - Captures the number of historical or missed searches after an outage, system restart, or data collection interruption.

    • Skipped detections: Captures the number of scheduled detections that were not run during the outage.

    • Recommended detections to rerun: Captures the number of detections to rerun after an outage.

  6. Select Skipped to view detections that were not run due to the outage during a specific time window.
    Note: Splunk Enterprise Security uses the Last 48 hours as the default time range to view skipped detections. However, if you are an administrator you can extend the time range to Last 7 days to view skipped detections.

  7. Review the color coded heat map in the right panel to view the detections that were run, skipped, and so on within the last 48 hours.

  8. Select the specific detections or bulk select all the detections that you want to rerun. A pop-up preview window is displayed when you rerun the detections.

  9. Select Rerun to rerun the detections. Detections that are rerun have a small icon displayed next to them and a message appears indicating that a rerun is in progress. The Backfill activity key indicator also displays the updated number of detections that are being rerun. You can use this indicator to prioritize and throttle backfill dispatch and restore coverage without impacting detection performance in your production environment.

  10. Go to Launchpad to view the status of all the detections and identify the detections that are queued or rerun.

  11. Go to Mission Control and select Analyst queue to view the findings that have an icon next to them, which indicates that the findings were created by the detections that were rerun.