Create a Monitoring MetricSet with a custom dimension

Learn how to create a Monitoring MetricSet with a custom dimension.

Create Monitoring MetricSets (MMS) with custom dimensions to monitor your services in real time. Splunk APM provides a default set of dimensions, but you can index additional dimensions using indexed span attributes. Use custom MMS to create custom charts, dashboards, and detectors to track what matters most to your team.

For more information about custom MMS, see About custom Monitoring MetricSets.

Prerequisites

To create a custom MMS, you must meet the following requirements:

  • You must have an admin role.

Before you create a custom MMS, you can optionally:

Create a custom Monitoring MetricSet for an instrumented service

Create a custom MMS for an instrumented service for more detailed monitoring and analysis.

Before you create a custom MMS for an instrumented service, first consider:

  • How you want to allocate the cardinality of your data to make your MMS most efficient.

The following table describes the configuration options for custom MMS. Select the best option for your needs to efficiently allocate your organization's cardinality entitlement. You can also filter by specific tag values to further reduce the cardinality of your MMS.

Option

Guidance

Service-level MMS and endpoint-level MMS for all endpoints within a service

This option creates a custom dimensionalized MMS for each selected service, as well as an MMS for each endpoint in each selected service. This is the highest cardinality option.

Service-level MMS and endpoint-level MMS for specific endpoints

This option creates a custom dimensionalized MMS for each selected service and an MMS for specific endpoints you select. In the box labeled Endpoint Filters, provide a list of endpoints or a regular expression pattern to create MMS for specific endpoints you’re interested in. This helps you to minimize unnecessary cardinality.

Service-level MMS only

This option creates an MMS for each selected service and no endpoint-level MMS. This is the lowest cardinality option.

Complete these steps to add a custom Monitoring MetricSet for an instrumented service.
  1. Navigate to the APM & RUM MetricSets page using one of the following methods:
    • From the Splunk Observability Cloud main menu, select Settings, then APM & RUM MetricSets.
    • In Splunk APM, select APM Configuration, then APM MetricSets.
  2. Select Add Custom MetricSet. The Add MetricSet panel displays.
  3. For Name, select a tag to add as a custom dimension to the MMS.
  4. The Scope determines how APM associates the span attribute or process with services in a trace. For Service enter the service(s) for which you want to create a Monitoring MetricSet.
    You can only create custom MMS for service-level indexed span attributes. You can't create custom MMS for globally indexed span attributes.
  5. Enable the Also create Monitoring MetricSet (MMS) option.
  6. In the Add tag data to drop-down menu, select how you want to add tag or process data to your Monitoring MetricSet. See the prerequisite of this section for more details.
  7. (Optional) In the Filter by tag values field, add tag filters to narrow the scope of your MMS to include only data associated with the tag values you enter. This can help further reduce cardinality in your MetricSets.
  8. Select Start Analysis to submit the configuration. Splunk APM runs an analysis of the span attribute or process to calculate the potential cardinality contribution of the MetricSet to determine whether it is within your entitlement limit.
    The Pending MetricSets section of the table displays your new MetricSet with an Analyzing status while the cardinality check runs. You might have to wait a few moments for the check to run.
    Note: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.
  9. After the cardinality check has completed running, review the cardinality and activate your new MetricSet:
    • If the cardinality impact of indexing this new MetricSet is within your subscription limit, the Status column displays a green check mark and Within Entitlement. Select the check mark under Actions to activate your Monitoring MetricSet.
    • If the check times out before you activate the MetricSet, run the analysis again.
    • If the check shows as failed, your cardinality is over your subscription limit for MMS or is otherwise invalid. See Cardinality contribution of indexed span attributes and processes. For steps to configure your MetricSet to be more efficient or other troubleshooting solutions, see Troubleshoot cardinality in Monitoring MetricSets.
    Note: For non-histogram MMS, Splunk APM replaces dots with underscores in dimension names for MMS time series. For histogram MMS, underscores are preserved.

Create a custom Monitoring MetricSet for an inferred service

Create a custom MMS for an inferred service for more detailed monitoring and analysis.

Note: Only 3rd-party or uninstrumented HTTP services are supported for MMS.
Complete the following steps to create a custom Monitoring MetricSet for an inferred service.
  1. Navigate to the APM & RUM MetricSets page using one of the following methods:
    • From the Splunk Observability Cloud main menu, select Settings, then APM & RUM MetricSets.
    • In Splunk APM, select APM Configuration, then APM MetricSets.
  2. Scroll to the Inferred Service Monitoring MetricSets section and select Add Inferred Service MMS. The Add Monitoring MetricSet for inferred service panel displays.
  3. For Inferred Service, select the inferred service for which you want to create a Monitoring MetricSet.
  4. Select Save.
    The Pending MetricSets section of the table displays your new MetricSet with an Analyzing status while the cardinality check runs. You might have to wait a few moments for the check to run.
    Note: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.
  5. After the cardinality check has completed running, review the cardinality and activate your new MetricSet:
    • If the cardinality impact of indexing this new MetricSet is within your subscription limit, the Status column displays a green check mark and Within Entitlement. Select the check mark under Actions to activate your Monitoring MetricSet.
    • If the check times out before you activate the MetricSet, run the analysis again.
    • If the check shows as failed, your cardinality is over your subscription limit for MMS or is otherwise invalid. See Cardinality contribution of indexed span attributes and processes. For steps to configure your MetricSet to be more efficient or other troubleshooting solutions, see Troubleshoot cardinality in Monitoring MetricSets.
    Note: For non-histogram MMS, Splunk APM replaces dots with underscores in dimension names for MMS time series. For histogram MMS, underscores are preserved.

Next steps

How to create charts, dashboards, and alerts with your custom Monitoring MetricSet.

Complete the following steps to create charts, dashboards, and alerts based on your custom Monitoring MetricSet.
  1. Create charts, dashboards, or alerts by following the relevant documentation for your task:

    Task

    Documentation

    Create charts

    Create charts in Splunk Observability Cloud

    Create dashboards

    Create and customize dashboards

    Create an alert

    Configure detectors and alerts in Splunk APM

    Monitor services in APM dashboards

    Track service performance using dashboards in Splunk APM

  2. In the above workflows, apply the filter sf_dimensionalized:true to filter out the metrics created by the default Monitoring MetricSet.
  3. In the above workflows, apply your custom MMS by filtering by the tag name that you indexed using the syntax sf_mms_id:<tag_name>