Learn about Monitoring MetricSets in Splunk APM
Learn about MetricSets in Splunk Observability Cloud. MetricSets are metrics for traces and spans in Splunk APM.
MetricSets are key performance indicators, like request rate, error rate, and request duration, that are calculated from traces and spans in Splunk APM. MetricSets are similar to the metric time series (MTS) used in Splunk Infrastructure Monitoring to populate charts and generate alerts. See Metric time series to learn more. MetricSets are MTS that are specific to Splunk APM.
There are 2 categories of MetricSets: Troubleshooting MetricSets (TMS), used for high-cardinality troubleshooting, and Monitoring MetricSets (MMS), used for real-time monitoring.
Monitoring MetricSets overview
Monitoring MetricSets (MMS) are MTS that power monitoring capabilities in Splunk APM, including:
Charts and dashboards.
The APM Overview page and the dashboard view.
Alerts. Detectors monitor MMS to generate alerts.
Splunk APM provides 5 default types of MMS, each with a set of dimensions such as deployment.environment and service.name. You can configure additional custom dimensions by indexing other span attributes.
MMS are created for spans where the span.kind has a value of SERVER or CONSUMER. Spans might lack a kind value, or have a different kind value, in the following situations:
The span originates in self-initiating operations or inferred services.
An error in instrumentation occurs.
Splunk Observability Cloud stores MMS for 13 months by default.
Available default MMS metrics and dimensions
MMS are available for the APM components listed in the following table. Each MMS also has a set of dimensions you can use to monitor and alert on service performance.
| Metric name | Dimensions | Custom dimension available (Yes/No)? |
|---|---|---|
service.request - the requests to endpoints in a service |
| Yes |
inferred.services - the requests to a service that has not yet been instrumented |
| No |
spans - the count of spans (a single operation) |
| Yes |
traces - the count of traces (collection of spans that represents a transaction) |
| No |
workflows - created by default when you create a business transaction |
| No |
Monitoring MetricSets in APM are generated as histogram metrics. Histogram metrics represent a distribution of measurements or metrics, with complete percentile data available. Data is distributed into equally sized intervals, allowing you to compute percentiles across multiple services, and aggregate data points from multiple metric time series.
Histogram metrics provide an advantage over other metric types when calculating percentiles, such as the p90 percentile for a single MTS. Learn more in Metric types. For histogram MMS, there is a single metric for each component.
Previously, MMS were classified as either a counter or gauge metric type. The previous MMS included 6 metrics for each component. The following table lists the previous MMS and current histogram MMS metrics.
Previous MMS and current histogram MMS metrics
Metric | Previous MMS | Histogram MMS (recommended) |
|---|---|---|
Request count |
|
|
Minimum request duration |
|
|
Maximum request duration |
|
|
Median request duration |
|
|
Percentile request duration |
|
|
Percentile request duration |
|
|
Example histogram metrics in APM
A histogram MTS uses the following syntax using SignalFlow: histogram(metric=[,filter=][,resolution=]) .
The following table displays example SignalFlow functions.
Example SignalFlow MMS functions
| Description | Previous MMS function | Histogram MMS function (recommended) |
|---|---|---|
| Aggregate count of all MTS | A = data('spans.count').sum().publish(label='A') | A = histogram('spans').count().publish(label='A') |
| P90 percentile for single MTS | filter_ = filter('sf_environment', 'environment1') and filter('sf_service', 'service 1') and filter('sf_operation', 'operation1') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = data('spans.duration.ns.p90', filter=filter_, rollup='sum').publish(label='A') | filter_ = filter('sf_environment', 'environment1') and filter('sf_service', 'service 1') and filter('sf_operation', 'operation1') and filter('sf_httpMethod', 'POST') and filter('sf_error', 'false') A = histogram('spans', filter=filter_).percentile(pct=90).publish(label='A') |
| Combined p90 for multiple services | A = data('service.request.duration.ns.p90', filter=filter('sf_service', 'service 2', 'service 1'), rollup='average').mean().publish(label='A') | A = histogram('service.request', filter=filter('sf_service', 'service 2', 'service1')).percentile(pct=90).publish(label='A') |
histogram(). To display all of the metric sets separately, each dimension must be applied as a group-by.About custom Monitoring MetricSets
You can add custom dimensions to MMS by indexing span attributes. Use custom MMS to filter and aggregate the MMS by specific indexed span attributes or processes such as cloud.provider and service.version.
You can create custom MMS at the service level and the endpoint or span level. When you create a custom dimension for a service-level MMS, APM creates an MMS that includes the service-level metrics with your chosen indexed span attribute or process as a custom dimension. If you add endpoint-level metrics, APM creates MMS that include span-level metrics, with your chosen indexed span attribute or process as a custom dimension.
To learn more about a specific scenario for custom MMS, see Scenario: Kai monitors detector service latency for a group of customers.
To learn how to create and use a custom MMS, see Create a Monitoring MetricSet with a custom dimension.
Scope of custom Monitoring MetricSets
You can create custom MMS for endpoints (span), services (service.request) and inferred services (inferred.services), but not for business transactions (workflow) or traces (trace) at this time. Custom MMS aren't supported for global tags. See Available default MMS metrics and dimensions.
Metrics and dimensions of custom Monitoring MetricSets
Each MMS has a set of metrics and dimensions for spans and traces you can use to monitor and alert on service performance. To prevent over counting metrics in aggregations, the built-in dashboards and charts in Splunk APM automatically exclude custom MMS. Custom MMS have a marker dimension, sf_dimensionalized: true, to include custom MMS.
When you create your dashboards and charts, you can exclude custom MMS by adding a filter on !sf_dimensionalized: true. If you want to look at the time series of a custom MMS in your charts, filter on sf_dimensionalized: true and then aggregate by the custom dimension or multi-dimension MMS ID you want to look at.
Cardinality contribution of indexed span attributes and processes
When you index a new span attribute or process to create custom MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span attribute or process. This gives you control of what you index and helps you to account for organization subscription limits.
If you try to index a span attribute or process that might increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags or processes for instrumented services by modifying or removing indexed span attributes or processes.
To see your TMS or MMS subscription limit, navigate to Settings, then Subscription usage. Depending on your organization subscription, you might need to go to Settings then Billing and usage. Select the APM tab and then select the Troubleshooting MetricSets or Monitoring MetricSets panel to view your subscription limit for MMS. You must have an admin or usage role to view subscription limits. To learn more about APM usage and billing, see Monitor Splunk APM billing and subscription usage.
Use MMS within Splunk APM
Use MMS for alerting and real-time monitoring in Splunk APM. You can create charts, dashboards, and alerts based on default and custom Monitoring MetricSets.
To use a default MMS, use one of the following workflows and filter by the default MMS name.
Task | Documentation |
|---|---|
Create charts | |
Create dashboards | |
Create an alert | |
Monitor services in APM dashboards |
To use a custom MMS, see the Next steps section of Create a Monitoring MetricSet with a custom dimension.