Analyze service performance with Tag Spotlight

Learn how to view metrics for every indexed span tag for a service in a single window in Splunk Observability Cloud.

Use Tag Spotlight to analyze the performance of your services in Splunk Observability Cloud to discover trends that contribute to high latency or error rates with span tags you have indexed. See Index span tags to create Troubleshooting MetricSets for more steps to index span tags.

For every service, Tag Spotlight provides time-series charts for request, error, and duration (RED) metrics for every value of each indexed span tag. Request and error charts display the total number of requests, errors, and root cause errors, and duration charts display p50, p90, and p99 latency. These values are based on Troubleshooting MetricSets (TMS), which Splunk APM generates for every indexed span tag. When you select span tag values or a time range, you can view relevant traces to learn more about outlying incidents.

The following image shows the a chart for requests and errors and a chart for duration for an example frontend service. Under the charts there are bar charts with RED metrics for each indexed span tag.

This image shows requests and errors and latency charts as well as requests and errors broken down by indexed span tag for an example frontend service.

To learn how to index span tags to generate Troubleshooting MetricSets, see Index span tags to create Troubleshooting MetricSets. For a general overview of MetricSets in APM, see Learn about MetricSets in APM.

View service performance by indexed span tags

To view service performance broken down by your indexed span tags, follow these steps:

  1. Go to Tag Spotlight. You can access Tag Spotlight in several places:

    • Select the Tag Spotlight panel on the APM landing page or in the service map.

    • Select the Tag Spotlight tab in the service view for your service.

    • Use the search in the top toolbar to search for Tag Spotlight and select the navigation result to go to Tag Spotlight.

  2. Add time range, environment, workflow, service, operation, and tag filters as need to refine the data in your Tag Spotlight view. The default time range is for the last 15 minutes, and the data resolution is 10 seconds.

  3. Use the request and errors and latency time-series charts at the top to see a distribution of your RED metrics.

  4. Use the bar charts under the request and errors and latency time-series charts to view RED metrics for each indexed span tag.

  5. View the distribution of all indexed span tags. The tag bar charts display either request and error distributions or latency distribution. Use the Cards display menu to select the data you want to display in the bars.

  6. Select the menu on the top left of the bar chart section to select which metrics to display in each tag panel. You can also use this menu to select whether to display tags with no values.

Customize tags display on Tag Spotlight

To configure the layout of the cards on the Tag Spotlight page, follow these steps:

  1. From the menu on the top left of the bar chart, select Customize card display order.

  2. Drag each of the span tags to arrange the order that cards are displayed on the page. Arrange the tags by priority, order of importance, or other criteria.

  3. Select Save.

To understand the source of an incident use 1 of these options to drill down into indexed span tags for a selected service:

  • Use the filter bar to filter the requests shown in Tag Spotlight by environment, service, and operation. You can also create a custom filter based on indexed or unindexed tags. The filter context is preserved even when you navigate away from the page.

  • Select the menu on the top left of the bar chart section to select which metrics to display for any indexed span tag.

  • Select a span tag value then select Add to filter to filter the RED metrics charts to spans containing that value.

  • Hover over the RED metrics charts to view a summary of metrics for the specified time. The resolution of data for the RED metrics chart is 10 seconds.

  • Select within the RED metrics charts to load example traces for the time you selected.

  • Select a span tag card header to add the top 5 span tag values in the card to the RED metrics charts.

  • Control what appears in the RED metrics chart by showing or hiding metrics in the chart’s legend. For example, you can select the eye icon beside Requests to view only errors.

Learn more

To view a detailed examples using Tag Spotlight, see: