Group related alerts with ITSI

Buttercup Bank, a fictitious company created for illustrative purposes in this scenario, has a database that acts as the main data source for their mobile banking application. Lauren, an IT administrator at Buttercup Bank, uses Splunk IT Service Intelligence (ITSI) with Splunk App for Content Packs and the Content Pack for Monitoring and Alerting to monitor the service health and key performance indicators (KPI) of their mobile banking application.

Group related alerts to reduce alert noise

To group related alerts generated by ITSI, Lauren does the following:

  1. Opens ITSI.
  2. Uses the entity discovery search function to verify that every active aggregation policy has a corresponding entity.
  3. Verifies that each of the following services are enabled:
    1. ITSI Event Analytics
    2. ITSI Alert Analytics
    3. ITSI Episode Analytics
  4. Selects Service Analyzer from the navigation bar.
  5. Applies filters for services, KPIs, and tags by selecting values for each of those fields.
  6. Selects Configuration > Notable Event Aggregation Policies from the navigation bar.
  7. Enables the following aggregation policies by toggling their switches in the Status column:
    1. Entity Type Alerts
    2. KPI Alerting Policy
    3. Normalized Policy

    Setting ITSI event aggregation policy

  8. Selects the aggregation policy in the Title column to configure episode information for that policy.
  9. Validates the output from the previous steps in this task by selecting Alerts and Episodes from the navigation bar. Lauren knows that notable events are stored in the itsi_tracked_alerts index. Alerts and Episodes now shows notable events grouped into deduplicated episodes in the ITSI Episode Review dashboard:

    ITSI Episode Review dashboard

Gain a holistic understanding of issues in the ITSI environment

Lauren can use aggregation policies in ITSI to group alerts logically for faster tracking and troubleshooting while viewing details in the Service Analyzer and Alerts and Episodes Review dashboards that are part of ITSI.

Lauren also knows that when Splunk Observability Cloud alerts are integrated with ITSI, a holistic understanding of Observability events, which are not the same as notable events tracked by ITSI, can help with problem diagnosis and troubleshooting.

Observability events provide context for point-in-time occurrences and display as overlays atop familiar dashboards, complete with event markers that show the start and end times of a given event.

Example of Observability events display

To access event data by creating an alert sent directly from ITSI to Splunk Observability Cloud and leverage the capability of event analytics in ITSI, Lauren does the following:

  1. Installs Splunk Observability Cloud Alert Action for Splunk.
  2. Pastes the Observability API token into the Configuration field of that app.
  3. Runs a correlation search.
  4. Creates an alert based on search results.
  5. Chooses fields such as app_name, eventtype, and detectorID from the search results to add as dimensions included with events.
  6. Adds the alert action for the app to take, such as specifying that email alias lauren@buttercupbank be designated as an alert recipient.

Summary

In this scenario, Lauren reduced alert noise by using aggregation policies to group related alerts in ITSI, and integrated ITSI with Splunk Observability Cloud so as to have similar visibility into Observability events. As a result, Lauren can more efficiently monitor system health and key performance indicators for Buttercup Bank, reducing mean time to problem detection and mean time to problem resolution.