Overview of the ITSI Rules Engine

The IT Service Intelligence (ITSI) Rules Engine is a system for continuously processing notable events to allow for event grouping and deduplication, as well as automatic action execution, based on user-defined criteria. The system revolves around a continuously running indexed real-time search that streams all notable events into a custom search command.

A notable event aggregation policy is the fundamental unit of the Rules Engine. Aggregation policies are the data structure the Rules Engine uses to group notable events into episodes. It's also the container for action rules that automate episode actions, such as sending an email or pinging a host. For more information about aggregation policies, see Overview of aggregation policies in ITSI.

Rules Engine search

The Rules Engine search is the component that groups notable events into episodes based on the filtering criteria you define in aggregation policies. The search is similar to the following: 

search `itsi_tracked_alerts_index` | itsi_rules_engine | where 1=2

The search runs with a time range of earliest=rt, latest=rt. It includes indexedRealTime=1 to force Splunk to stream all newly indexed events directly to the custom search command itsi_rules_engine. The final WHERE clause ensures the output of the search command doesn't linger in the dispatch directory.

How the Rules Engine functions

The Rules Engine's functionality begins with correlation searches. Correlation searches generate notable events in ITSI, which are stored in the itsi_tracked_alerts index. The Rules Engine saved search accepts notable events into the itsi_rules_engine custom search command. The search command generates the internal structures required for aggregating events into episodes and executing actions.

Episodes are stored via the HTTP Event Collector (HEC) in the itsi_grouped_alerts index. All episode metadata is stored via REST in the itsi_notable_group_system and itsi_notable_group_user KV store collections.

The Rules Engine search periodically polls the configuration database for updates. If a policy indicates some action should be executed, the Rules Engine dispatches a REST request to the Event Management Interface to execute the action. For more information, see Event Management Interface in the REST API Reference manual.

The following diagram illustrates the Rules Engine workflow of aggregating events, storing episodes, and executing actions:

RE.png

Sort notable events with the Rules Engine

When incoming notable events are generated within a few seconds of each other and you want to process them according to the event time stamp, update the Rules Engine using these steps:

If you are using correlation searches to generate notable events, update the following:

  1. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add: rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY
  2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_event_management.conf and add: [tracked_alert] sort_notable_events = 1
  3. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf and add: [itsi_event_generator] param.is_use_event_time = 1
  4. Restart Splunk.

If you are directly ingesting notable events through HEC, update the following:

  1. Ensure that the events are ingested in a sorted manner according to the timestamp in an ascending order.
  2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add: rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY
  3. Restart the Rules Engine.
  4. Navigate to Activity, then Jobs, then Set all filters to All, then search for label="itsi_event_grouping", then stop the job. Wait for a few minutes to Rules Engine to run again.