Send alert notifications to Splunk On-Call using Splunk Observability Cloud

Configure Splunk Observability Cloud to send alerts to Splunk On-Call when a detector alert condition is met and when the condition clears.

You can configure Splunk Observability Cloud to automatically send alert notifications to Splunk On-Call (formerly VictorOps) when a detector alert condition is met and when the alert clears.

To send Splunk Observability Cloud alert notifications to Splunk On-Call, complete the following configuration tasks:

Step 1: Get your Splunk On-Call service API endpoint URL

You must be a Splunk On-Call global admin or alert admin to complete this task.

To get the service API endpoint URL in Splunk On-Call:

  1. Log in to Splunk On-Call.

  2. Select the Integrations tab.

  3. Select the 3rd Party Integrations tab.

  4. Select the Splunk Observability Cloud System Monitoring tile. The Service API Endpoint value displays.

    If you don’t see an endpoint URL value, select Enable Integration to generate one.

  5. Copy the entire endpoint URL, including the $routing_key text, for use in Step 3: Create a Splunk On-Call integration in Splunk Observability Cloud.

Step 2: Get your Splunk On-Call alert routing key

You must be a Splunk On-Call global admin or alert admin to complete this task.

For information about how to get your Splunk On-Call alert routing key, see Create Routing Keys in Splunk On-Call.

You’ll need your alert routing key in Step 3: Create a Splunk On-Call integration in Splunk Observability Cloud.

Step 3: Create a Splunk On-Call integration in Splunk Observability Cloud

You must be a Splunk Observability Cloud administrator to complete this task.

To create a Splunk On-Call integration in Splunk Observability Cloud:

  1. Log in to Splunk Observability Cloud.

  2. Open the Splunk On-Call guided setup . Optionally, you can navigate to the guided setup on your own:

    1. In the left navigation menu, select Data Management.

    2. Go to the Available integrations tab, or select Add Integration in the Deployed integrations tab.

    3. In the integration filter menu, select All.

    4. In the Search field, search for Splunk On-Call, and select it.

    5. Select New Integration to display the configuration options.

  3. By default, the name of the integration is VictorOps. Give your integration a unique and descriptive name. For information about the downstream use of this name, see About naming your integrations.

  4. In the Post URL field, enter the service API endpoint URL value you copied from Splunk On-Call in Step 1: Get your Splunk On-Call service API endpoint URL.

  5. Save.

  6. If Splunk Observability Cloud can validate the Splunk On-Call service API endpoint URL, a Validated! success message displays. If you get an error, make sure that the URL value you entered matches the value displayed in Splunk On-Call in Step 1: Get your Splunk On-Call service API endpoint URL.

Step 4: Add a Splunk On-Call integration as a detector alert recipient in Splunk Observability Cloud

To add a Splunk On-Call integration as a detector alert recipient in Splunk Observability Cloud:

  1. Create or edit a detector that you want to configure to send alert notifications using your Splunk On-Call integration.

    For more information about working with detectors, see Create detectors to trigger alerts and Subscribe to alerts using the Detector menu.

  2. In the Alert recipients step, select Add Recipient.

  3. Select VictorOps and then select the name of the Splunk On-Call integration you want to use to send alert notifications. This is the integration name you created in Step 3: Create a Splunk On-Call integration in Splunk Observability Cloud.

  4. Enter the routing key you got in Step 2: Get your Splunk On-Call alert routing key

  5. Activate and save the detector.

Splunk Observability Cloud sends an alert notification to your Splunk On-Call timeline when an alert is triggered by the detector and when the alert clears.

Splunk Observability Cloud alert notification fields sent to Splunk On-Call

Here are the Splunk Observability Cloud alert notification fields that are sent to Splunk On-Call.

Field

Description

Detector Definition

Displays a link to view the Splunk Observability Cloud detector and corresponding alert rules.

Graph

Displays a snapshot view of the signal that triggered the alert.

detector

Displays the name of the Splunk Observability Cloud detector.

inputs

Displays detailed information about the Splunk Observability Cloud alert, including the rule and detector names, alert triggering conditions, and signal details.

rule

Displays the name of the Splunk Observability Cloud alert rule where the conditions to trigger and clear alert events are defined.

entity_display_name

Displays the Splunk Observability Cloud rule and detector name. This information also appears in the rule and detector notification fields.

state_message

When the alert is triggered, displays the alert’s severity. Valid values include: critical, major, minor, warning, or info. When the alert is resolved, displays the alert’s resolution. Valid values include: back to normal, stopped, or manually resolved.

entity_id

Displays the incident’s ID.

monitoring_tool

Displays signalfx.

message_type

Displays the alert’s severity. Valid values include: critical, warning, acknowledgement, info, or recovery.