Learn about Troubleshooting MetricSets in Splunk APM

Troubleshooting MetricSets (TMS) are metric time series (MTS) you can use for troubleshooting high-cardinality identities in APM.

MetricSets are key performance indicators, like request rate, error rate, and request duration, that are calculated from traces and spans in Splunk APM. MetricSets are similar to the metric time series (MTS) used in Splunk Infrastructure Monitoring to populate charts and generate alerts. See Metric time series to learn more. to learn more. MetricSets are MTS that are specific to Splunk APM.

There are 2 categories of MetricSets: Troubleshooting MetricSets (TMS), used for high-cardinality troubleshooting, and Monitoring MetricSets (MMS), used for real-time monitoring.

Troubleshooting MetricSets overview

Troubleshooting MetricSets (TMS) are metric time series (MTS) you can use for troubleshooting high-cardinality identities in APM. You can also use TMS to make historical comparisons across spans and business transactions.

Splunk APM indexes and generates Troubleshooting MetricSets for several span attributes by default. For more details about these span attributes, see Default indexed span tags in APM.

You can't modify or stop APM from indexing the default span attributes, but you can create custom TMS by indexing additional span attributes and processes. See Index span tags to create Troubleshooting MetricSets to learn how.

Splunk Observability Cloud retains TMS for the same amount of time as raw traces. By default, the retention period is 8 days.

Available TMS metrics

Every TMS creates the following metrics, known as request, error, and duration (RED) metrics. RED metrics appear when you select a service in the service map.

  • Request rate

  • Error rate

  • Root cause error rate

  • p50, p90, and p99 latency

The measurement precision of Troubleshooting MetricSets is 10 seconds. Splunk APM reports quantiles from a distribution of metrics for each 10-second reporting window.

Cardinality contribution of indexed span attributes and processes

When you index a new span attribute or process to create custom MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span attribute or process. This gives you control of what you index and helps you to account for organization subscription limits.

If you try to index a span attribute or process that might increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags or processes for instrumented services by modifying or removing indexed span attributes or processes.

To see your TMS or MMS subscription limit, navigate to Settings, then Subscription usage. Depending on your organization subscription, you might need to go to Settings then Billing and usage. Select the APM tab and then select the Troubleshooting MetricSets or Monitoring MetricSets panel to view your subscription limit for MMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see Monitor Splunk APM billing and subscription usage.