Requirements

Supported Splunk versions:

• Splunk Enterprise, Splunk Cloud Platform Version: 9.0*, 8.2*, 8.1*, 8.0*, 7.3

• Python version 2 or 3 is supported

* There is an outstanding issue affecting the calendar display on certain Splunk versions.

General requirements:

• You need an active Splunk On-Call instance before you begin.

• On-Premises

  • Open port 443 for outgoing https communication with Splunk On-Call.

  • Proxy is supported (available with 1.0.5 version and higher) from the Splunkbase. See Splunk Add-on for On-Call (VictorOps) on Splunkbase.

Setup

1. Download the add-on from the Splunkbase here.

2. Go to Apps > Manage Apps > Install App From File and import the downloaded .tar.gz file.

3. The Splunk add-on for On-Call is now be visible as an app in Splunk.

4. Go to the app.

5. Under Inputs, select Create New Input and choose the type of data you want Splunk to ingest from Splunk On-Call.

6. Enter the following configuration details:

   • Index - select the Splunk index. Select 1 index per input. Update the dashboard search macros to use the index name for data to populate the dashboards.

   • Organization ID - Enter the Splunk On-Call organization the data is coming from. This is important if you are collecting from multiple organizations in Splunk On-Call.

   • API ID - Splunk On-Call admins or alert admins can find your API ID in Splunk On-Call under Integrations > API.

   • API Key - Splunk On-Call admins or alert admins can find your API Key in Splunk On-Call under Integrations > API.

Input options

There are 4 types of inputs collected: users, oncall, teams (which includes routing keys) and incidents. Select the inputs you want to index per organization. See the following list for inputs and their attributes in a sample JSON format.

Users (type=user)

• Info

  • Names (first, last, username)

  • Created date

  • Date created

  • Date password updated

  • Verified

• Contact Methods – Name, verification status (phone only) and value of all contact methods.

• Paging Policy

• Organization

On-Call (type=oncall, events are split per team)

• Organization

• Team name, slug

• Escalation Policy

  • Oncall user(s) at time of index

Teams (type=team)

• Info

  • Number of members, verified members

  • Team name, slug

• Members

  • Username, first name, last name

  • Verified

• Organization

• Policies

  • Name, slug

Routing Keys (type=routingkey)

• Default routing key status (true/false)

• Organization

• Name

• Target escalation policies

  • Escalation policy name, slug

  • Team name, slug

Incidents (source=victorops_incidents)

• Paged Users, Teams

• State changes (ack, resolve)

• All Metadata

• Index timestamp is set to the startTime field

• Alert Count

Webhook setup

While the webhook configuration is available if needed, we recommend the native add-on configuration.

Ingest data through a webhook

Splunk On-Call sends data to Splunk using an HTTP Endpoint Collector (HEC) depending upon your deployment a heavy forwarder might also be needed. To ensure communication from Splunk On-Call to Splunk, allow the Splunk On-Call range of IP addresses.

When setting up the HEC in Splunk, create a new Source Type for the type of data that you're sending in. This allows you to send in and keep track of multiple different types of OnCall data like chats, incident action logs, different team incidents, etc.

Create webhooks

Create 4 outgoing webhooks, 1 for each even type. See the following configuration details.

The body of each webhook varies according to the event-type. Replace your organization ID found in the URL of Splunk On-Call For example, https://portal.victorops.com/dash/org_ID/outgoing-webhooks) in all instance of org_ID.

Event Type: Any Incidents

Body:

{
 "sourcetype": "_json",
 "event":
 {
 "slug": "org_ID",
 "link": "https://portal.victorops.com/client/org_ID/popoutIncident?incidentName=${{STATE.INCIDENT_NAME}}",
 "type": "incident",
 "alertService": "${{ALERT.service}}",
 "hostName": "${{ALERT.host_name}}",
 "service": "${{ALERT.service}}",
 "ENTITY_TYPE": "${{INCIDENT.ENTITY_TYPE}}",
 "SERVICESTATE": "${{ALERT.SERVICESTATE}}",
 "VO_ALERT_RCV_TIME": "${{ALERT.VO_ALERT_RCV_TIME}}",
 "alert_url": "${{ALERT.alert_url}}",
 "entity_display_name": "${{ALERT.entity_display_name}}",
 "entity_state": "${{ALERT.entity_state}}",
 "message_type": "${{ALERT.message_type}}",
 "monitor_name": "${{ALERT.monitor_name}}",
 "monitoring_tool": "${{ALERT.monitoring_tool}}",
 "routing_key": "${{ALERT.routing_key}}",
 "alert_timestamp": "${{ALERT.timestamp}}",
 "ACK_MSG": "${{STATE.ACK_MSG}}",
 "ACK_USER": "${{STATE.ACK_USER}}",
 "ACK_TIMESTAMP": "${{STATE.ACK_TIMESTAMP}}",
 "ALERT_COUNT": "${{STATE.ALERT_COUNT}}",
 "CURRENT_ALERT_PHASE": "${{STATE.CURRENT_ALERT_PHASE}}",
 "CURRENT_STATE": "${{STATE.CURRENT_STATE}}",
 "ENTITY_ID": "${{STATE.ENTITY_ID}}",
 "IncidentNum": "${{STATE.INCIDENT_NAME}}",
 "INCIDENT_TIMESTAMP": "${{STATE.INCIDENT_TIMESTAMP}}",
 "LAST_TIMESTAMP": "${{STATE.LAST_TIMESTAMP}}",
 "MONITOR_TYPE": "${{STATE.MONITOR_TYPE}}",
 "stateService": "${{STATE.SERVICE}}",
 "alert_uuid": "${{ALERT.VO_UUID}}"
 }
}

Event Type: Any Paging

Body:

{
 "sourcetype": "_json",
 "event":{ 
 "slug":"org_ID", "type":"paging", "user": "${{PAGE.USER_ID}}", "started":"${{PAGE.STARTED}}", "page_id": "${{PAGE.ID}}", "attempt_num": "${{PAGE.ATTEMPT_NUMBER}}", "method_type": "${{PAGE.METHODS.0.TYPE}}", "method_label": "${{PAGE.METHODS.0.LABEL}}", "cancellation": "${{PAGE.CANCELLATION}}" } }

Event Type: Any On-Call

Body:

{
 "sourcetype": "_json",
 "event":{ 
 "slug":"org_ID",
 "type":"oncall",
 "user":"${{ONCALL.USER_ID}}",
 "state":"${{ONCALL.STATE}}",
 "team":"${{ONCALL.TEAM_NAME}}",
 "group":"${{ONCALL.GROUP_ID}}",
 }
}

Event Type: All Chats

Body:

{
 "sourcetype": "_json",
 "event":{ 
 "slug":"org_ID",
 "type":"chat",
 "user": "${{CHAT.USER_ID}}",
 "text": "${{CHAT.TEXT}}",
 "is_robot": "${{CHAT.IS_ROBOT}}"
 }
}

Troubleshooting

Verify the following if you encounter trouble: s

1. Check that the API credentials are correct. Use the public API key and ID found under Integrations > API. Do not use the API key found under Integrations > Splunk Enterprise.

2. Check that the host can reach the Splunk On-Call API. Try running 'ping api.victorops.com' to confirm the connection.

3. You can investigate further by inspecting the logs in $SPLUNK_HOME/var/log/splunk/ta_splunk_add_on_for_victorops_victorops_inyput_type.log.

4. If you are polling incidents in an organization with more than 60 incidents in the past 7 days, the incident poll can take some time to run due to Splunk On-Call API rate limits. If the input has been configured correctly and incident data is still not appearing, check the above log path for the incidents log (for example, tail -f ta_splunk_add_on_for_victorops_victorops_incidents.log), if the last log entry is similar to "Waiting 59.985822999999996 seconds", the script is waiting on rate limits to finish collecting and indexing the data. If this issue persists, consider reducing the polling interval.

5. If dashboard items are not appearing, check the dashboard macros under Settings > Advanced Search > Search macros to ensure the index name you created for the inputs is being used in the macros.