Use webhooks in escalation policies

Use webhooks to specify callbacks from Splunk On-Call escalation policies to your applications.

Webhooks are a way to specify callbacks from Splunk On-Call to your own applications, and can be added to your teams’ escalation policies in order to receive incident details and process them however you wish. For more detail about escalation policies, see Set up an escalation policy.

Some examples of how these could be used:

Automatically bounce a server process when there’s an incident related to it.
Integrate Splunk On-Call incidents into your own service dashboard.
Keep a status page up-to-date with outages

Webhook content type: application/x-www-form-urlencoded

Get started

To get started, from the Splunk On-Call timeline navigate to Integrations and select Webhooks.

On the integrations page, select webhooks.

You are prompted for a Name and URL for the webhook. Once submitted, an Auth Code will be generated.

Authenticate webhook requests

When you create a webhook, we generate a secure, random authentication token. POST requests are signed with this key, so you can verify the incoming request, to make sure that it actually came from Splunk On-Call.

In order to authenticate that the POST requests are arriving to your application from Splunk On-Call complete these steps:

Create a string with the URL of the webhook, exactly how it appears in Splunk On-Call; this includes trailing slashes.
Sort the request’s POST variables alphabetically by key.
Append each POST variable’s key and value to the URL string, with no delimiter.
Create a binary hash of the resulting string with MAC-SHA1, using the webhook’s authentication key.
Base64 encode the binary signature.
Compare the output with the key X-VictorOps-Signature in the request - if it matches, the request originated from Splunk On-Call (formerly VictorOps).

SSL certificate chain verification

If your webhook uses a secure connection (i.e. the URL starts with "https://"), the receiving endpoint must use a valid SSL certificate signed by a recognized Certificate Authority.

This recommendation was implemented based on SSL Labs Best Practices .

Webhook escalation steps

When a webhook is part of a team’s escalation policy, your service will receive an HTTP POST request when the incident is escalated. The request will contain the following information:

POST Body:

Incident: the identifier of the incident in Splunk On-Call.
Summary: a short description of the incident.
Message: a message about the incident.

HTTP Headers

X-Victorops-Signature: a signature based on the auth key of the webhook in Splunk On-Call (formerly VictorOps).

Once there is a webhook, it can be added to an escalation policy.

Add the webshook to the escalation policy.

Difference between escalation and custom outgoing webhooks

For more information on custom outgoing webhooks, see Custom outbound webhooks in Splunk On-Call.

Sample scala code for reference

import javax.crypto.Mac
import javax.crypto.spec.SecretKeySpec import javax.xml.bind.DatatypeConverter
def generateSignature(key: String, url:String, postData: Map[String, String]) = {
   val contents =postData.toList.sorted.foldLeft(url) { case (s, (key, value)) =>s"\ :math:`s`\ key$value" }
   val mac = Mac.getInstance("HmacSHA1")
   mac.init(new SecretKeySpec(key.getBytes, "HmacSHA1"))
   DatatypeConverter.printBase64Binary(mac.doFinal(contents.getBytes("utf-8")))
}

Documentation