Create a Monitoring MetricSet with custom dimensions

Learn how to create a Monitoring MetricSet with up to 5 custom dimensions.

Create Monitoring MetricSets (MMS) with up to 5 custom dimensions to monitor your services in real time. Splunk APM provides a default set of dimensions, but you can index additional dimensions using indexed span attributes. Use custom MMS to create custom charts, dashboards, and detectors to track what matters most to your team.

For more information about custom MMS, see About custom Monitoring MetricSets.

Prerequisites

To create a custom MMS, you must meet the following requirements:

  • You must have an admin role.

Before you create a custom MMS, you can optionally:

Create a custom Monitoring MetricSet for an instrumented service

Create a custom MMS for an instrumented service for more detailed monitoring and analysis.

Before you create a custom MMS for an instrumented service, first consider:

  • How you want to allocate the cardinality of your data to make your MMS most efficient.

The following table describes the configuration options for custom MMS. Select the best option for your needs to efficiently allocate your organization's cardinality entitlement. You can also filter by specific tag values to further reduce the cardinality of your MMS.

Option

Guidance

Service-level MMS and endpoint-level MMS for all endpoints within a service

This option creates a custom dimensionalized MMS for each selected service, as well as an MMS for each endpoint in each selected service. This is the highest cardinality option.

Service-level MMS and endpoint-level MMS for specific endpoints

This option creates a custom dimensionalized MMS for each selected service and an MMS for specific endpoints you select. In the box labeled Endpoint Filters, provide a list of endpoints or a regular expression pattern to create MMS for specific endpoints you’re interested in. This helps you to minimize unnecessary cardinality.

Service-level MMS only

This option creates an MMS for each selected service and no endpoint-level MMS. This is the lowest cardinality option.

Complete these steps to add a custom Monitoring MetricSet for an instrumented service.
  1. Navigate to the APM & RUM MetricSets page using one of the following methods:
    • From the Splunk Observability Cloud main menu, select Settings > APM & RUM MetricSets.
    • In Splunk APM, select APM Configuration > APM MetricSets.
  2. Select Add MetricSet Configuration > Add Custom MetricSet. The Add Custom MetricSet panel displays.
    Note: If you have already indexed the span or process, it appears in the list of Custom MetricSets and is already generating Troubleshooting MetricSets. If you select the edit icon, you can only add or update an MMS with a single dimension.
  3. For Tag selection, select 1-5 tags to add as custom dimension(s) to the MMS.
    For more information on using MMS with multiple custom dimensions, see About custom Monitoring MetricSets.
  4. The Scope determines how APM associates the span attribute or process with services in a trace. For Services in scope, enter the service(s) for which you want to create a Monitoring MetricSet.
    You can only create custom MMS for service-level indexed span attributes. You can't create custom MMS for globally indexed span attributes.
  5. Enable the Create Monitoring MetricSet (MMS) option.
  6. If you selected more than one tag to add as custom dimensions, provide a unique ID for your multi-dimension MMS in the Multi-dimension MMS ID (sf_mms_id) field. You'll filter with this multi-dimension MMS ID when you create charts, dashboards, and alerts.
  7. In the Add tag data to drop-down menu, select how you want to add tag or process data to your Monitoring MetricSet. See the prerequisite of this section for more details.
  8. In the Tag values to index drop-down menus, select the tag values to index to control your cardinality.
  9. Select Start Analysis to submit the configuration. Splunk APM runs an analysis of the span attribute or process to calculate the potential cardinality contribution of the MetricSet to determine whether it is within your entitlement limit.
    The Pending MetricSets section of the table displays your new MetricSet with an Analyzing status while the cardinality check runs. You might have to wait a few moments for the check to run.
    Note: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.
  10. After the cardinality check has completed running, review the cardinality and activate your new MetricSet:
    • If the cardinality impact of indexing this new MetricSet is within your subscription limit, the Status column displays a green check mark and Within Entitlement. Select the check mark under Actions to activate your Monitoring MetricSet.
    • If the check times out before you activate the MetricSet, run the analysis again.
    • If the check shows as failed, your cardinality is over your subscription limit for MMS or is otherwise invalid. See Cardinality contribution of indexed span attributes and processes. For steps to configure your MetricSet to be more efficient or other troubleshooting solutions, see Troubleshoot cardinality in Monitoring MetricSets.
    Note: For non-histogram MMS, Splunk APM replaces dots with underscores in dimension names for MMS time series. For histogram MMS, underscores are preserved.

Create a custom Monitoring MetricSet for an inferred service

Create a custom MMS for an inferred service for more detailed monitoring and analysis.

Note: Only 3rd-party or uninstrumented HTTP services are supported for MMS.
Complete the following steps to create a custom Monitoring MetricSet for an inferred service.
  1. Navigate to the APM & RUM MetricSets page using one of the following methods:
    • From the Splunk Observability Cloud main menu, select Settings > APM & RUM MetricSets.
    • In Splunk APM, select APM Configuration > APM MetricSets.
  2. Scroll down to the Inferred Service Monitoring MetricSets section and select Add Inferred Service MMS. The Create an inferred service Monitoring MetricSet panel displays.
    A screenshot of the Create an inferred service Monitoring MetricSet panel on the APM & RUM MetricSets page.
  3. For Inferred Service, select the inferred service for which you want to create a Monitoring MetricSet.
  4. (Optional) To create an MMS with custom dimensions from indexed span attributes:
    1. Enable the Configure custom tag indexing option.
    2. For Tag Selection, select 1-5 tags.
    3. For Monitoring MetricSet configuration, select the tag values to index to control your cardinality.
    For more information on using MMS with multiple custom dimensions, see About custom Monitoring MetricSets.
  5. Select Save.
    The Pending MetricSets section of the table displays your new MetricSet with an Analyzing status while the cardinality check runs. You might have to wait a few moments for the check to run.
    Note: Your pending MetricSet expires after 1 hour. If your MetricSet times out before you activate it, run the analysis again.
  6. After the cardinality check has completed running, review the cardinality and activate your new MetricSet:
    • If the cardinality impact of indexing this new MetricSet is within your subscription limit, the Status column displays a green check mark and Within Entitlement. Select the check mark under Actions to activate your Monitoring MetricSet.
    • If the check times out before you activate the MetricSet, run the analysis again.
    • If the check shows as failed, your cardinality is over your subscription limit for MMS or is otherwise invalid. See Cardinality contribution of indexed span attributes and processes. For steps to configure your MetricSet to be more efficient or other troubleshooting solutions, see Troubleshoot cardinality in Monitoring MetricSets.
    Note: For non-histogram MMS, Splunk APM replaces dots with underscores in dimension names for MMS time series. For histogram MMS, underscores are preserved.

Next steps

How to create charts, dashboards, and alerts with your custom Monitoring MetricSet.

Complete the following steps to create charts, dashboards, and alerts based on your custom Monitoring MetricSet.
  1. Create charts, dashboards, or alerts by following the relevant documentation for your task:

    Task

    Documentation

    Create charts

    Create charts in Splunk Observability Cloud

    Create dashboards

    Create and customize dashboards

    Create an alert

    Configure detectors and alerts in Splunk APM

    Monitor services in APM dashboards

    Track service performance using dashboards in Splunk APM

  2. In the above workflows, apply the filter sf_dimensionalized:true to filter out the metrics created by the default Monitoring MetricSet.
  3. In the above workflows, apply your custom MMS using one of the following methods:
    • If your MMS only has one custom dimension, filter by the tag name that you indexed using the syntax sf_mms_id:<tag_name>.
    • If your MMS has more than one custom dimension, filter by the multi-dimension MMS ID using the syntax sf_mms_id:<MMS_ID>. The APM & RUM MetricSets page displays MMS IDs in the MMS (MMS ID) column.