Send alert notifications to ServiceNow using Splunk Observability Cloud

Configure Splunk Observability Cloud to send alerts to ServiceNow when a detector alert condition is met and when the condition clears.

You can configure Splunk Observability Cloud to automatically send alert notifications to ServiceNow when a detector alert condition is met and when the alert clears.

To send Splunk Observability Cloud alert notifications to ServiceNow, complete the following configuration tasks:

Step 1: Choose the type of ServiceNow issue for your integration

Before you set up the integration, choose a ServiceNow issue type from the following table:

Issue type

Role required

ServiceNow endpoint

Problem

web_service_admin

/api/now/v2/table/problem

Incident

web_service_admin

/api/now/v2/table/incident

Event

evt_mgmt_integration, only if Requires ACL authorization is selected for Inbound Event Default Bulk Endpoint in Scripted Rest APIs. To learn more, see the ServiceNow support article on events.

/api/global/em/jsonv2

Import setweb_service_admin/api/now/v2/table/{CUSTOM_TABLE}

Make note of the role and receiving endpoint that corresponds to your issue type before proceeding with Step 2: Create a ServiceNow user or OAuth application for your Splunk Observability Cloud integration.

Note: The user_admin role is used to verify that ServiceNow has successfully created a Problem or Incident. The itil role is used to create Problems and Incidents when alerts are sent.

Step 2: Create a ServiceNow user or OAuth application for your Splunk Observability Cloud integration

To receive alert notifications from Splunk Observability Cloud, you can create a ServiceNow user or register Splunk Observability Cloud as an OAuth application within your ServiceNow instance, which allows ServiceNow to securely authenticate requests from Splunk Observability Cloud.
Note: You must be a ServiceNow administrator to complete this task.
If you already have OAuth configured in ServiceNow or an existing ServiceNow user that you want to use to receive alert notifications, the user has the roles assigned that correspond to your issue type, and you know the user ID and password, you can skip to Step 3: Create a ServiceNow integration in Splunk Observability Cloud.

To set up a ServiceNow user for your Splunk Observability Cloud integration:

  1. Log in to ServiceNow.

  2. In the left navigation panel, scroll to User Administration and select Users.

  3. Select New.

  4. Enter User ID, First name, and Last name values that clearly communicate that the user is associated with Splunk Observability Cloud notifications. Make note of the User ID value for use in subsequent steps.

  5. Enter a Password value. Make note of this value for use in Step 3: Create a ServiceNow integration in Splunk Observability Cloud.

  6. Select the Active check box.

  7. Select Submit.

  8. Find your new user by either searching for the user ID or doing a reverse chronological sort on the Created column. Select the user ID to open the user information window. Scroll down and select the Roles tab. Select Edit.

  9. In the Collection search field, enter the roles for the issue type you chose in Step 1: Choose the type of ServiceNow issue for your integration, for example, web_service_admin. Select the role and select > to move it the Roles List panel.

  10. Select Save. The new roles display on the Roles tab for the user.

To register Splunk Observability Cloud as an OAuth application in your ServiceNow instance:

  1. Follow configuration instructions on the ServiceNow documentation. For the following fields on the configuration page, you need to provide the information as suggested:

    FieldValue
    NameEnter a descriptive name for the application. For example, Splunk Observability Cloud.
    Redirect URL

    Enter https://api.{REALM}.signalfx.com/v2/integration/servicenow/oauth-callback. If your realm is us0, you don't need to include a realm in the URL.

    Access Token Lifespan

    Set a time between 30 to 60 minutes. Enter the value in seconds. For example, enter 1800 for an access token lifespan of 30 minutes.

    Refresh Token Lifespan

    Set a time between 3 to 6 months. Enter the value in seconds. For example, enter 8640000 for a refresh token lifespan of about 3 months.

  2. After configuring your OAuth application, you will see your new application on the Inbound Integrations page.

  3. Retrieve the necessary credentials to set up a ServiceNow integration in Splunk Observability Cloud:

    1. Select the new application to open the information window.

    2. Copy and save the value in the Client ID field.

    3. Select the lock icon next to the Client Secret field to see the value hidden by default. Copy and save this value in a secure place.

Step 3: Create a ServiceNow integration in Splunk Observability Cloud

You must be a Splunk Observability Cloud administrator to complete this task.

To create a ServiceNow integration in Splunk Observability Cloud:

  1. Log in to Splunk Observability Cloud.

  2. Open the ServiceNow guided setup. Optionally, you can navigate to the guided setup on your own:

    1. In the left navigation menu, select Data Management.

    2. Go to the Available integrations tab, or select Add Integration in the Deployed integrations tab.

    3. In the integration filter menu, select All.

    4. In the Search field, search for ServiceNow, and select it.

    5. Select Add Integration to display the configuration options.

  3. Give your integration a unique and descriptive name. For information about the downstream use of this name, see About naming your integrations.

  4. In the Authorization menu, select one of the following options and enter the information accordingly:

    Authorization option

    What to enter

    Username and password

    - In the Username field, enter the username from ServiceNow in Step 2: Create a ServiceNow user or OAuth application for your Splunk Observability Cloud integration.

    - In the Password field, enter the password from ServiceNow in Step 2: Create a ServiceNow user or OAuth application for your Splunk Observability Cloud integration.

    OAuth- In the Client ID field, enter the client ID associated with the application set up in ServiceNow in

    Step 2: Create a ServiceNow user or OAuth application for your Splunk Observability Cloud integration.

    - In the Client Secret field, enter the client secret generated when you create the application in ServiceNow in Step 2: Create a ServiceNow user or OAuth application for your Splunk Observability Cloud integration.

  5. In the Instance Name field, enter your ServiceName instance name. For example, the instance name must use the format example.service-now.com. Do not include a leading https:// or a trailing /. Additionally, you cannot use local ServiceNow instances.

    To troubleshoot potential blind server-side request forgeries (SSRF), Splunk Observability Cloud has included \*.service-now.com on an allow list. As a result, if you enter a domain name that is rejected by Splunk Observability Cloud, contact support to update the allow list of domain names. See Support Programs for support contact options.

  6. Select Incident, Problem, Event, or Import set to indicate the issue type you want the integration to create in ServiceNow. If necessary, you can create a second integration using another issue type. This lets you create an incident issue for one detector rule and a problem issue for another detector rule.

  7. Select Next.

  8. If Splunk Observability Cloud can validate the ServiceNow username, password, and instance name combination, a Validated! success message displays. If an error displays instead, make sure that the values you entered match the values in ServiceNow.

  9. On the Customize message page, you see the default template for a ServiceNow integration. You can customize your payload to make sure responders have the context needed to resolve the issues.

    For a full list of supported variables and examples, see Integrate ServiceNow with Splunk Observability Cloud in the Splunk Developer Portal.

  10. Select Next.

  11. Review your integration and select Save.

Note: After configuring your ServiceNow integration, you can manually refresh token by selecting Re-authorize from the action menu in the list view.

Step 4: Add a ServiceNow integration as a detector alert recipient in Splunk Observability Cloud

To add a ServiceNow integration as a detector alert recipient in Splunk Observability Cloud:

  1. Create or edit a detector that you want to configure to send alert notifications using your ServiceNow integration.

    For more information about working with detectors, see Create detectors to trigger alerts and Subscribe to alerts using the Detector menu.

  2. In the Alert recipients step, select Add Recipient.

  3. Select ServiceNow and then select the name of the ServiceNow integration you want to use to send alert notifications. This is the integration name you created in Step 3: Create a ServiceNow integration in Splunk Observability Cloud.

  4. Activate and save the detector.

Splunk Observability Cloud sends an alert notification to create an incident in ServiceNow when the detector triggers an alert. When the alert clears, it sends a notification that sets the incident state to Resolved.

For Incident and Problem issues, the ServiceNow integration sets the Impact and Urgency fields on the ServiceNow issue based on the Splunk Observability Cloud alert severity (see Severity). When you clear alerts for Problem and Incident issues, Splunk Observability Cloud marks them as Resolved.

The following table shows the Splunk Observability Cloud severity for Incident and Problem issues:

Splunk Observability Cloud severityServiceNow Impact and Urgency fields
Critical 1
Major or Minor 2
Warning or Info 3

For Event issues, the ServiceNow integration sets the Severity of the issue based on the Splunk Observability Cloud alert severity (see Severity). The Event integration also creates an event whenever an alert is sent or cleared.

The following table shows the Splunk Observability Cloud severity for Event issues:

Splunk Observability Cloud severity

ServiceNow Severity field

Clear

0

Critical

1

Major

2

Minor

3

Warning

4

Info

5