Troubleshoot missing notable events in Splunk Enterprise Security

Note: The documentation for Splunk Enterprise Security versions 8.0 and higher have been rearchitected from previous versions, causing some links to have redirect errors. For documentation on version 8.0, see Splunk Enterprise Security 8.x documentation.

If you have a Correlation Search that isn't generating notable events when you think it should, you can check the following potential causes and solutions.

Cause Solution
The notable events are being suppressed. Check to see if the notable index contains notable events. Search in Splunk Web against the notable index to determine if the notable event exists but is being excluded from Incident Review:

index=notable

Suppressions filter notable events from appearing in Incident Review. If you see your notable event in the index, then make sure that no suppressions are preventing the notable event from appearing in Incident Review.
The entire correlation search doesn't match, but part of it does. Run the correlation search manually over the given timeframe and see if it matches the events. If it doesn't match, remove parts of the search until you isolate the part of the search that doesn't match.
The notable alert action isn't triggered. Check the notable alert action logs. These logs indicate if the notable alert action is triggered to make a notable event. Search in Splunk Web to view these logs:

index=_internal sourcetype=notable_modalert
Splunk Enterprise cannot parse the stash file. Verify that the search output doesn't include any unnecessary output. Make sure that the correlation search only outputs the fields you really need, and that the fields don't include extra content such as XML or excessive amounts of text. Extra content can make it difficult for Splunk to parse the stash file. If the stash file can't be parsed, then your notable events may not be generated correctly.
The correlation search schedule is incorrect, not running, or suppressed. Check the search scheduler logs. Search in Splunk Web to view the scheduler logs:

index=_internal sourcetype=scheduler

Look for the following:
  • Make sure that the search is running during the time-frame that you expect events
  • See if suppressed indicates that events are suppressed
  • See if result_count indicates that notable events are created, for example, is greater than one
  • Check the status field to make sure that the search is running successfully
If you are using a distributed architecture, you may have missed creating the notable index on your cluster. See Configure and deploy indexes in the Installation and Upgrade Manual.