Set up Datadog alerts in ITSI

Integrate Datadog with Splunk IT Service Intelligence (ITSI) to send alerts between Datadog and ITSI.

Prerequisites

  • A Datadog account with the ability to create webhook integrations and edit monitors

  • A Splunk HEC endpoint and token (e.g., https://<splunk-host>:8088/services/collector)

  • Splunk HEC token with write access to the target index, and copy the generated token value:

    • Name: Datadog

    • Source type: datadog:webhook

    • Index: your target index (for example, main)

Note: If your Splunk HEC uses HTTPS with a self-signed certificate, you may need to disable SSL verification in Datadog or add the certificate to as a trusted certificate.

Datadog webhook setup

  1. In Datadog, navigate to Integrations, then search for Webhooks.
  2. Select the Webhooks tile, then select Configuration (or New Webhook if this is your first).
  3. Select + New to add a new webhook.
  4. Fill in the following fields:
    Field Value
    Name splunk_hec (this becomes the reference name — you will use @webhook-splunk_hec in monitors)
    URL https://<splunk-host>:8088/services/collector
  5. Paste the following into the Custom Headers field:
    JSON 
    {
  "Authorization": "Splunk <your-hec-token>",
  "Content-Type": "application/json"
}
  6. Replace the default payload with the following JSON. This payload uses the dd_ prefix namespace to avoid collisions with ITSI internal field names:
    JSON 
    {
  "dd_event_id": "$ID",
  "dd_aggreg_key": "$AGGREG_KEY",
  "dd_alert_cycle_key": "$ALERT_CYCLE_KEY",
  "dd_date": "$DATE",
  "dd_date_posix": "$DATE_POSIX",
  "dd_last_updated": "$LAST_UPDATED",
  "dd_event_type": "$EVENT_TYPE",
  "dd_event_title": "$EVENT_TITLE",
  "dd_event_msg": "$EVENT_MSG",
  "dd_text_only_msg": "$TEXT_ONLY_MSG",
  "dd_org_id": "$ORG_ID",
  "dd_org_name": "$ORG_NAME",
  "dd_alert_id": "$ALERT_ID",
  "dd_alert_status": "$ALERT_STATUS",
  "dd_alert_transition": "$ALERT_TRANSITION",
  "dd_alert_metric": "$ALERT_METRIC",
  "dd_alert_priority": "$ALERT_PRIORITY",
  "dd_alert_query": "$ALERT_QUERY",
  "dd_alert_scope": "$ALERT_SCOPE",
  "dd_alert_title": "$ALERT_TITLE",
  "dd_alert_type": "$ALERT_TYPE",
  "dd_hostname": "$HOSTNAME",
  "dd_tags": "$TAGS",
  "dd_link": "$LINK",
  "dd_snapshot": "$SNAPSHOT"
}
    Note: Datadog automatically substitutes all $VARIABLE placeholders at send time. For a full list of available variables, see the Datadog Webhooks documentation.

  7. Leave Encode as form unchecked. The payload must be sent as raw JSON for Splunk HEC.

  8. Select Save.

Add webhook to a monitor

Datadog fires webhooks when they are referenced in a monitor's notification message.

  1. Navigate to Monitors then Manage Monitors.

  2. Select an existing monitor (or create a new one).

  3. In the Notify your team section, add the webhook reference to the notification body:

    @webhook-splunk_hec

  4. Select Save.

You can add @webhook-splunk_hec to as many monitors as you like. Each time the monitor triggers, recovers, or changes state, Datadog will POST the webhook payload to your Splunk HEC endpoint.

Test Datadog alert

On the Webhooks configuration page, select the Test button next to your splunk_hec webhook.

Datadog will send a test payload with sample values. Confirm a test event appears in Splunk by running:

index=main sourcetype="dynatrace:problem" | head 10
When a Datadog event triggers, you will see an event similar to the following:
JSON 
{
  "dd_event_id": "5765793488018070528",
  "dd_aggreg_key": "",
  "dd_alert_cycle_key": "abc123def456",
  "dd_date": "1714300800",
  "dd_date_posix": "1714300800",
  "dd_last_updated": "1714300800",
  "dd_event_type": "query_alert_monitor",
  "dd_event_title": "High CPU on webserver01.example.com",
  "dd_event_msg": "CPU utilization has exceeded 90% on webserver01.example.com for the last 5 minutes.
Current value: 95.3%
Threshold: 90%",
  "dd_text_only_msg": "CPU utilization has exceeded 90% on webserver01.example.com for the last 5 minutes. Current value: 95.3% Threshold: 90%",
  "dd_org_id": "123456",
  "dd_org_name": "My Organization",
  "dd_alert_id": "56789012",
  "dd_alert_status": "Triggered",
  "dd_alert_transition": "Triggered",
  "dd_alert_metric": "system.cpu.user",
  "dd_alert_priority": "P2",
  "dd_alert_query": "avg(last_5m):avg:system.cpu.user{host:webserver01} > 90",
  "dd_alert_scope": "host:webserver01.example.com",
  "dd_alert_title": "[Triggered] High CPU on webserver01.example.com",
  "dd_alert_type": "error",
  "dd_hostname": "webserver01.example.com",
  "dd_tags": "env:production,service:web,team:infra",
  "dd_link": "https://app.datadoghq.com/monitors/56789012?to_ts=1714300800&from_ts=1714297200",
  "dd_snapshot": "https://p.datadoghq.com/snapshot/abc123.png"