Index span tags to create Troubleshooting MetricSets

Learn how to use index tags to create Troubleshooting MetricSets that help you troubleshoot services with Splunk Observability Cloud.

Use span tags or processes to break down services and inter-service calls along trace characteristics or attributes. To get additional value from a span tag or process, a Splunk APM administrator can run an action known as indexing, which activates additional analysis of the indexed span tag or process. One benefit of indexing is to get aggregated metrics, called MetricSets, across all spans that contain a specified indexed tag or process.

Prerequisites

To index span tags to create Troubleshooting MetricSets in Splunk APM, you must have the admin role.

Before indexing span tags and processes, you can optionally:

See Learn about Troubleshooting MetricSets for an overview of TMS concepts.
See Reference and best practices for span tags and Troubleshooting MetricSets for guidance on choosing span tags or processes to index.

Index a new span tag or process

Follow these steps to index a span tag or process to create a Troubleshooting MetricSet.

To get to the APM MetricSets page, do one of the following:
1. From APM > Overview, select APM Configuration > APM MetricSets from the menu. The APM MetricSets page opens.
2. From anywhere in Splunk Observability Cloud, select Settings in the navigation bar and select APM MetricSets under Product settings.
On the APM & RUM MetricSets page, select Add Custom MetricSet.
Enter the Name of a span tag or process you want to index.
The Scope determines how APM associates the span tag or process with services in a trace:
- Select Service to associate the span tag or process with services. This means the value of the span tag or process might change across services in a given trace. Select All Services to index the span tag or process for every service. Select specific services to index the span tag or process for only those services.
  Note: If you index a span tag or process at the service level, you can also use it as a custom dimension in Monitoring MetricSets. See Create a Monitoring MetricSet with a custom dimension for instructions.
- Select Global to associate the span tag or process with traces. This means the value of the span tag or process is the same for all services in a given trace.
For more information about scope, see Types of span tags you can index.
Select Start Analysis to submit the configuration. Splunk APM runs an analysis of the span tag or process to calculate the potential cardinality contribution of the MetricSet to determine whether it is within your entitlement limit.
Your new metric set shows under Pending MetricSets section of the table with an Analyzing status while the cardinality check runs. You might have to wait a few moments for the check to run.
CAUTION: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.
After the cardinality check has run, review the cardinality and activate your new MetricSet.
- If the cardinality impact of indexing this new MetricSet is within your subscription limit, the Status column displays a green checkmark and Within Entitlement. Select the check mark under Actions to activate your Troubleshooting MetricSet.
- If the cardinality check times out before you activate the MetricSet, run the analysis again.
- If the cardinality check shows as failed, your cardinality is over your subscription limit for TMS. See Cardinality contribution of indexed span tags and processes.
The following example shows a cardinality check.