Removing redundant alerts with the dedup command

Alert throttling, while helpful, can create excessive notifications due to redundant risk events stacking up in the search results. In such cases, use the dedup command to make sure that each event counts only once toward the total risk score. Using the dedup command in the logic of the risk incident rule can remove duplicate alerts from the search results and display only the most recent notifications prior to calculating the final risk score.

For example, use the dedup command to filter the redundant risk notables by fields such as risk_message, risk_object, or threat_object.

| dedup risk_message risk_object threat_object

Remove redundant alerts with the dedup command

Follow these steps to use the dedup command in the risk incident rule to remove redundant alerts:

Note: The following procedure is for illustrative purposes only. Fields and values that you use might vary.
  1. Navigate to the base search of the risk incident rule. 
    BY All_Risk.risk_object,All_Risk.risk_object_type
| 'drop_dm_object_name("All_Risk")'
  2. Use the streamstats command to retain the original score, sources, and risk messages. 
    BY All_Risk.risk_object,All_Risk.risk_object_type
| 'drop_dm_object_name("All_Risk")'
| streamstats sum(risk_score) as original_score values(source) as sources
values(risk_message) as risk_messages
BY risk_object
  3. Use the eval command and the case function to identify the risk messages that might inflate the risk score. The following search creates a new field called adjust_score that you can use to combine the risk events (i.e. risk messages) if they match the stated criteria. If there is no match, the field adjust_score is empty. 
    | eval adjust_score = case(
source IN ("My Noisy Rule That Fires a Lot but I Still Want to Know About,
Once", "My Other Really Useful Context Low Risk Rule"),"1",
match(risk_message,"IDS - Rule Category 1.*|IDS - Rule Category 2.*") OR
match(risk_message,"DLP - Rule Category 1.*|DLP - Rule Category 2.*"),"1",
1=1,null())
  4. Use the coalesce function to take the new field, which just holds the value "1" if it exists. If it does not exist, use the risk message. 
    | eval combine = coalesce(adjust_score,risk_message)
  5. Use the dedup command on the combine field to add the results from the saved fields so that the noisy alerts are counted only once. 
    | dedup combine risk_score
| streamstats sum(risk_score) as risk_score values(sources) as source
values(risk_messages) as risk_message
BY risk_object

In summary, use the following search to remove the redundant alerts:

BY All_Risk.risk_object,All_Risk.risk_object_type | `drop_dm_object_name("All_Risk")` | streamstats sum(risk_score) as original_score values(source) as sources values(risk_message) as risk_messages BY risk_object | eval adjust_score = case( source IN ("My Noisy Rule That Fires a Lot but I Still Want to Know About, Once", "My Other Really Useful Context Low Risk Rule"),"1", match(risk_message,"IDS - Rule Category 1.*|IDS - Rule Category 2.*") OR match(risk_message,"DLP - Rule Category 1.*|DLP - Rule Category 2.*"),"1", 1=1,null()) | eval combine = coalesce(adjust_score,risk_message) | dedup combine risk_score | streamstats sum(risk_score) as risk_score values(sources) as source values(risk_messages) as risk_message BY risk_object