Accelerate troubleshooting using Business Workflows on Tag Spotlight

This Splunk APM example describes how to use APM Tag Spotlight of Business Workflows to accelerate troubleshooting.

Deepu, the payment service owner at Buttercup Games, wants to understand how errors in the payment service impact business KPIs.

To understand the impact of errors on business KPIs, Deepu takes the following steps:

  1. Deepu opens Tag Spotlight and selects the affected service

  2. Deepu reviews the errors by workflow

  3. Deepu filters Tag Spotlight to find the subset of traces where most errors occur

  4. Deepu opens an example trace

  5. Deepu switches to Splunk Log Observer for further troubleshooting

Deepu opens Tag Spotlight and selects the affected service

Deepu uses search to search for Tag Spotlight and selects the navigation result to go to Tag Spotlight. They select the paymentservice.

This animation shows the user searching for Tag Spotlight and selecting the payment service in the Tag Spotlight filter.

Deepu reviews the errors by workflow

Deepu reviews the errors by workflow and notices that the requests for the frontend:/cart/checkout workflow return a majority of root-cause errors.

This screenshot shows the Tag Spotlight view of the payment service, which shows that the frontend:/cart/checkout workflow has the most errors

Deepu filters Tag Spotlight to find the subset of traces where most errors occur

Because APM supports infinite cardinality, Deepu adds the frontend:/cart/checkout workflow to the filter and applies additional filters for version to v350.10 and Kind to SERVER to examine the subset of traces where the most errors occur:

This animated gif shows the steps to add filters for frontend:/cart/checkout workflow filtered by Version and Kind in Tag Spotlight.

Deepu opens an example trace

Because Splunk APM stores all traces without sampling, Deepu selects the peak of errors on the chart to see example traces of those tag combinations at that timestamp. Deepu finds that the requests from checkoutservice to paymentservice are failing.

This screenshot shows an example trace with errors in Tag Spotlight.

Deepu switches to Splunk Log Observer for further troubleshooting

Deepu remembers that version 350.10 of the payment service recently deployed a backward-incompatible change to its API. Because Deepu turned on Related Content in Splunk APM, Deepu can select Logs for payment service to switch to Splunk Log Observer to investigate whether the deployment causes the error.

This screenshot shows Tag Spotlight with the option to select to view Logs for paymentservice highlighted.

Summary

Using Business Workflows and Tag Spotlight Deepu quickly isolated a trace with an error. Because Deepu had related content turned on, Deepu was able to quickly navigate from the trace to logs to further investigate causes of the errors.

Learn more