Requirements

• SolarWinds Orion NPM is required for the integration

• SAM is required for the acknowledge-back component

Configure in Splunk On-Call

1. In your account, go to Integrations then SolarWinds.

2. Activate the integration.

3. Copy the Service API Endpoint to use in later steps.

Configure in SolarWinds

1. Go to Alerts & Activity then Alerts then select Manage Alerts to go to the Alert Manager.

2. Select Add New Alert

3. Name the alert "Send Alert to Splunk On-call".

4. Configure your trigger, reset, and time-of-day conditions.

5. On the Trigger Actions tab, select Add Action.

6. Select Send a Get or Post a URL to a Web Server and select Configure Action.

7. Name your action "Critical to Splunk On-Call".

8. Select Use HTTP/S POST and enter your Splunk On-Call SolarWinds Service API Endpoint. Replace the trailing $routing_key with the actual routing key you intend to use for the alert. For more information on routing keys, see Create Routing Keys in Splunk On-Call.

9. In Body to POST, copy and paste an applicable Trigger Action payload from the example payloads. See Example body payloads.

10. Set Content Type to application/json.

11. Set Authentication to None.

12. Configure the rest of the settings as needed and save your changes.

13. Select Next to open the Reset Action tab.

14. Select Add Action.

15. In Body to POST copy and paste the corresponding Reset Action payload from the example payloads. See Example body payloads.

16. Select Add Action.

17. Select Next through the Summary tab, review, then Submit.

Alert Splunk On-Call for multiple actions

1. Go to Alerts & Activity then Alerts then select Manage Alerts to go to the Alert Manager.

2. Select the box next to the newly-created "Send Alert to Splunk On-Call" alert and select Assign Action.

3. Assign your trigger and reset actions, then select Assign.

Activate the Splunk On-Call acknowledge back

This acknowledge back is a command poll against the Splunk On-Call public API, configured in SolarWinds as an Application Manager. You have to create a new user and node in SolarWinds.

1. Go to Main Settings & Administration in SolarWinds.

2. Go to Product Specific Settings then SAM Settings.

3. Go to Application Monitors then Manage Application Monitors then Application Monitor Templates.

4. Import the VictorOps Ack-Back.apm-template and replace the script with the SolarWinds-ack-back-v2.1.txt.

   • VictorOps Ack-Back.apm-template

   • SolarWinds-ack-back-v2.1.txt

5. From the Main Settings & Administration section in SolarWinds, go to Product Specific Settings then SAM Settings

6. Go to Application Monitors then Manage Application Monitors.

7. Select VictorOps Ack-Back and select Edit Properties. The Polling Frequency is set to 60seconds, which means SolarWinds command-polls Splunk On-Call every 60 seconds for new acknowledgement.

8. Expand the component Ack-Back Get Requests.

9. Change the Credential for Monitoring to the SolarWinds API User.

10. Review the Script Body to confirm it includes the v2 version of the ack-back script.

11. Update the PowerShell script with your Splunk On-Call API values and SolarWinds primary server name. You can find the Splunk On-Call API values from your Splunk On-Call under Settings then API. Near the top of the PowerShell script replace everything between the quotes with the correct information for the following variables:

    • #Victor Ops Company ID

    • $API_ID="<Your API ID>"

    • #Victor Ops API Key

    • $ApiKey="<Your API Key>"

    • #Set to your SolarWinds Primary Application Server

    • $SolarWindsServer="<Your Primary SolarWinds Server Name>"

12. Select Submit.

Create a limited-rights user for SolarWinds alert management

1. Go to Settings then All Settings to reach the Main Settings & Administration section in SolarWinds.

2. Go to User Accounts then Manage Accounts.

3. Select Add New Account.

4. Select Orion individual account.

5. Select Next.

6. In the User Name field enter "solarwindsapi".

7. Enter a password.

8. Select Next.

9. In the Define Settings section, scroll to Alerts.

10. In Allow Alert Management Rights select Yes.

11. Select Submit.

Add the limited-rights user to the SAM credential library

1. From the Main Settings & Administration section in SolarWinds, go to Product Specific Settings then SAM Settings.

2. Go to Global SAM Settings then Credentials Library.

3. Add the limited user account. For the credential name use SolarWinds API User.

Create an external node for VictorOps.com

1. Go to Settings then Manage Nodes.

2. Add a node.

3. For the node name enter "victorops.com".

Assign the Splunk On-Call acknowledge back application templates

1. From the Main Settings & Administration section in SolarWinds, go to Product Specific Settings then SAM Settings.

2. Go to Application Monitors then Manage Application Monitors.

3. Select Application Monitor Templates.

4. Select VictorOps Ack-Back then select Assign to Node.

5. Select the "victorops.com" node and select the green arrow to add it to Selected Nodes.

6. Select Next.

7. Assign the SolarWinds API User credentials.

8. Select Assign Application Monitors.

Example body payloads

The following template payloads contain the required fields, you can add fields as needed.

The backslash \ character can’t be used in the HTTP post. Instead use an SQL replace function in the SolarWinds Alert:

SQL: SELECT REPLACE (''‘${Caption}''‘,'\‘,' ')}

Node is down

Trigger action

{
  "message_type":"CRITICAL",
  "monitor_name":"SolarWinds",
  "monitoring_tool":"SolarWinds",
  "alert_rule":"${N=Alerting;M=AlertName}",
  "state_message":"${NodeName} is ${Status}",
  "entity_display_name":"${NodeName} is ${Status}",
  "entity_id":"${N=Alerting;M=AlertObjectID}",
  "host_name":"${NodeName}",
  "ip_address":"${Node.IP_Address}"
}

Reset action

{
  "message_type":"RECOVERY",
  "monitor_name":"SolarWinds",
  "monitoring_tool":"SolarWinds",
  "alert_rule":"${N=Alerting;M=AlertName}",
  "state_message":"${NodeName} is ${Status}",
  "entity_display_name":"${NodeName} is ${Status}",
  "entity_id":"${N=Alerting;M=AlertObjectID}",
  "host_name":"${NodeName}",
  "ip_address":"${Node.IP_Address}"
}

Volume space alert

Trigger action

{
  "alert_rule":"${N=Alerting;M=AlertName}",
  "entity_display_name":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",
  "entity_id":"${N=Alerting;M=AlertObjectID}",
  "host_name":"${NodeName}",
  "ip_address":"${Node.IP_Address}",
  "message_type":"CRITICAL",
  "monitor_name":"SolarWinds",
  "monitoring_tool":"SolarWinds",
  "state_message":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"
}

Reset action

{
  "alert_rule":"${N=Alerting;M=AlertName}",
  "entity_display_name":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free",
  "entity_id":"${N=Alerting;M=AlertObjectID}",
  "host_name":"${NodeName}",
  "ip_address":"${Node.IP_Address}",
  "message_type":"RECOVERY",
  "monitor_name":"SolarWinds",
  "monitoring_tool":"SolarWinds",
  "state_message":"${NodeName} ${SQL: SELECT REPLACE ('''${Caption}''','\',' ')} has ${VolumeSpaceAvailable} free"
}

Component-based alert

Trigger action

{
  "message_type":"CRITICAL",
  "monitor_name":"SolarWinds",
  "monitoring_tool":"SolarWinds",
  "alert_rule":"${N=Alerting;M=AlertName}",
  "state_message":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
  "entity_display_name":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
  "entity_id":"${N=Alerting;M=AlertObjectID}",
  "host_name":"${NodeName}",
  "ip_address":"${Node.IP_Address}"
}

Reset action

{
  "message_type":"RECOVERY",
  "monitor_name":"SolarWinds",
  "monitoring_tool":"SolarWinds",
  "alert_rule":"${N=Alerting;M=AlertName}",
  "state_message":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
  "entity_display_name":"${NodeName} ${N=SwisEntity;M=ComponentAlert.ComponentName} is ${N=SwisEntity;M=Status;F=Status}",
  "entity_id":"${N=Alerting;M=AlertObjectID}",
  "host_name":"${NodeName}",
  "ip_address":"${Node.IP_Address}"
}