Set up

Add users

The most important first step of setting up Splunk On-Call is adding users. There are two main methods to add new users:

1. Leverage their email address in the portal (Select Users, then Invite User.)

2. Use our API (ID and Key required, which can be found by selecting Integrations then API)

Create teams

Teams are made up of:

• User lists

• On-call shifts

• Escalation policies

To create a team, navigate to the Teams section from the top navigation bar. From the Teams page, select Add Team, then choose a name.

We recommend standardizing your team names to clearly delineate across teams. You can choose team names based on service, internal team name — whatever makes sense to your organization.

Invite users and declare admins

Once you’ve built a few teams, the next step is to add people. You can add invited users. Then establish a hierarchy of users based on user roles, for example: Admins, Users, and Team Admins.

Create rotations

Create escalation policies

Escalation policies determine which incidents are routed, to whom they are routed, and how they are escalated. Essentially, an escalation policy is how Splunk On-Call escalates a triggered event.

A best practice for setting up your escalation policy is to establish a minimum of three escalation paths: on-duty user, previous/next user in a rotation, and manager/team lead.

Configure routing keys

Routing keys tie the alerts from your monitoring tools to the specific team (or escalation policy) in Splunk On-Call. This helps get the right person on the problem and reduce alert noise for those unrelated to a specific incident. These can be found by selecting Settings, then Routing Keys.

Use the name of the team or policy that is handling the alerts, the service or host for the alert, monitoring tool the alert is coming from. Although routing keys are case insensitive, we recommend using all lowercase letters to prevent alerts from going to the default routing team.

• Matching team name: CloudOps (team) = cloudops (routing key)

• Matching monitoring tool: Splunk (tool) = splunk (routing key)

Integrations

The final piece is to set up your custom integrations. Integrations will feed alerts into Splunk On-Call in order to create incidents which will then page out.

We recommend setting up any chat integrations or non-alerting integrations before setting up your alerting integrations.

Rules engine

The Rules Engine is a Full-Stack service level feature. It is a rules engine that allows you to set certain conditions and trigger custom actions such as annotating alerts with images/links/notes, overwrite alert fields or add new fields, when those conditions are met.

Report on team activity and performance

As an admin, it is important that you are able to track and report out on team activity and performance in order to continuously improve. Navigate to the Reports page in the top navigation menu.

Post-incident review

Gain historical insight on incidents and a documented account of how you solved the problem.

• Performance (MTTA/MTTR) Report: Tells the story of your investment in Splunk On-Call and the practice of DevOps.

• On-Call Report: Take a look into time spent on-call and number of incidents worked by team/user.

• Incident Frequency Report: Analyze the flow of incidents after the fact so you can go upstream to solve the incident causing the problem in your system.

Adjust license numbers

If you ever need to significantly increase or decrease your Splunk On-Call license numbers, please reach out to your Regional Sales Manager or Customer Success Manager. If you are unsure of who to engage, please send your inquiry to victorops-sales@splunk.com.