Link APM services, traces, and spans to relevant resources

An overview of global data links for services, traces, and spans to other resources.

Prerequisite

To create global data links, you must have the admin role.

The following task describes how to create a global data link for a service that links to an Infrastructure Monitoring navigator. You can set up a global data link for any service or a specific service.

  1. In the Splunk Observability Cloud main menu, select Settings then Global Data Links.

  2. Select New Link.

  3. Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific service.

  4. For Show On, select Any Value of and enter sf_service to associate the global data link with every service. If you want to create the global data link for a specific service, select Property:Value Pair and enter sf_service:<yourServiceName> for the service you want to create the global data link for.

  5. If you want the global data link to display based on the Show On value and 1 or more additional conditions, select Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.

  6. Select Choose Navigator and select the navigator you want to associate with the global data link.

  7. Select Save. When you view a service that matches the Show On value, you can carry the entire context of the service to the navigator.

You can create a global data link that passes information about APM services, traces, and spans to custom dashboards available in Splunk Observability Cloud. When you select a global data link to an Infrastructure Monitoring dashboard, the entire context of the property you were viewing transfers to the dashboard. For example, if you were viewing a service, the global data link transfers information about any endpoints you filtered for, the selected environments, and any tags you filtered for within the selected time range to the dashboard.

The following task describes how to create a global data link for a service that links to an Infrastructure Monitoring dashboard. You can set up a global data link for any service or a specific service.

  1. In the Splunk Observability Cloud main menu, select Settings then Global Data Links.

  2. Select New Link.

  3. Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific service. For example, you might enter Trace Ingest Dashboard.

  4. For Show On, select Any Value of and enter sf_service to associate the global data link with every service. If you want to create the global data link for a specific service, select Property:Value Pair and enter sf_service:<yourServiceName> for the service you want to create the global data link for.

  5. If you want the global data link to display based on the Show On value and 1 or more additional conditions, select Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.

  6. Select Choose Dashboard and select the dashboard you want to associate with the global data link.

  7. Select Save. When you view a service that matches the Show On value, you can carry the entire context of the service to the dashboard.

You can create a global data link that passes information about an APM service, trace, or span to a Splunk Cloud Platform or Splunk Enterprise search. This means you can create a global data link that runs a Splunk search query to parse logs for any service, trace ID, or span ID you’re analyzing.

Create a global data link for each APM property that you want to connect to logs in a Splunk instance. For example, if you want to connect a trace ID to a logs in a Splunk instance, create a global data link that carries the trace ID context to the Splunk instance. To connect a span ID to logs in a Splunk instance, create a global data link that carries the span ID context to the Splunk instance.

The following task describes how to create a global data link for a trace ID. The global data link runs a Splunk search query for log events that include a specific trace ID in a Splunk instance. The process is the same for creating a global data link for a service, span ID, or span tag: use a Show On value for the property you want to create a global data link for.

  1. In the Splunk Observability Cloud main menu, select Settings then Global Data Links.

  2. Select New Link.

  3. Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific trace ID. For example, you might enter Splunk Cloud Platform Search.

  4. For Link to, select Splunk.

  5. For Show On, select Any Value of and enter trace_id.

  6. If you want the global data link to display based on the Show On value and one or more additional conditions, select Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.

  7. Enter the Splunk instance fully qualified domain name (FQDN) and port of your Splunk instance for the URL. For example, you might enter https://<yourHostname>.splunkcloud.com:443 for a Splunk Cloud Platform instance. You might also specify the FQDN and port for a Splunk Enterprise instance you have access to.

  8. Keep the Minimum Time Window at 1m.

  9. If your Splunk instance refers to fields differently than APM refers to them, associate APM fields with related fields in Splunk. For example, sf_service in APM could be service in your Splunk instance. If something like this is the case, specify the Splunk Observability Cloud Term value and map it to an External Term value.

  10. Select Save. When you view a specific trace, you can drill down into this global data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.

You can create a global data link that passes information about an APM service, trace, or span to a Kibana URL. By passing APM properties and their characteristics in a Kibana URL, you can transfer context from Splunk Observability Cloud to Kibana.

The following task describes how to create a global data link for a log filter in Kibana for a selected trace ID. You can also filter on other APM properties: use a Show On value for the property you want to create a global data link for.

  1. Open the Splunk Observability Cloud main menu.

  2. Hover over Data Configuration and select Global Data Links.

  3. Select New Link.

  4. Enter a Link Label. This is what you select when you want to use the global data link to drill down into a specific trace ID. For example, you might enter Kibana filter.

  5. For Link to, select Kibana.

  6. For Show On, select Any Value of and enter trace_id.

  7. If you want the global data link to display based on the Show On value and one or more additional conditions, select Add Conditions. To define a condition based on a property name, select Any Value of and enter a property name. To define a condition based on a property name and a specific value, select Property:Value Pair and enter a property name and value pair. If you define multiple conditions, all conditions must be met for the link to display.

  8. Enter a Kibana URL that includes the trace_id field in a log filter for the URL. For example, you can enter a URL like this one: http://<yourKibanaFQDN>/kibana/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'{{start_time}}',mode:absolute,to:'{{end_time}}'))&_a=(columns:!(_source),interval:auto,query:(language:kuery,query:'traceId:{{value}}'),sort:!('@timestamp',desc))

  9. Enter your preferred Time Format.

  10. Keep the Minimum Time Window at 1m.

  11. If Kibana refers to fields differently than APM refers to them, associate APM fields with related fields in Kibana. For example, sf_service in APM could be service in Kibana. If something like this is the case, specify the Splunk Observability Cloud Term and External Term. If the field names are the same, skip this step.

  12. Select Save. When you view a specific trace, you can drill down into this global data link and view a Splunk search that includes all log events with the trace ID within the time range of the trace.

Note: You can only create a global data link from a Splunk APM inferred service to a Splunk AppDynamics tier if the tier is monitored by a Splunk AppDynamics SaaS environment.

This section describes how to create global data links to Splunk AppDynamics tiers using the user interface. To programmatically create global data links with Terraform, see Manage global data links to Splunk AppDynamics tiers with Terraform.

  1. In the Splunk Observability Cloud main menu, select Settings > Global Data Links.

  2. Select New link.

  3. Enter a Link label. This is what you select when you want to use the global data link to drill down into a specific Splunk AppDynamics tier. For example, you might enter Splunk AppDynamics tier.

  4. For Link to, select AppDynamics tier.

  5. For Show on, select an inferred service from the list. For more information on inferred services, see Analyze the performance of inferred services.

  6. For URL, enter the Splunk AppDynamics tier URL. To obtain the tier URL, navigate to the tier in the Splunk AppDynamics UI and copy the URL from the browser. Ensure that you capture the entire URL and that it contains the controller URL, application ID, and application component.

  7. Select Save. When you view an inferred service in the APM Service Map, you can select the data link to navigate to the service in Splunk AppDynamics.

Transfer APM context in a custom URL

You can create a global data link that passes information about an APM service, trace, or span to a custom URL.

For example, you can specify a custom URL like this to transfer the context of a service to a URL: https://www.example.com/search/?field={{key}}&value={{value}}&service={{properties.sf_service}}&st={{start_time}}&et={{end_time}}.

Learn more