Splunk APM Business Workflows

Splunk APM Business Workflows can be imported into ITSI using the Splunk Observability - Import Business Workflows Configuration Dashboard. The APM topology for the configured Business Workflow is used to define the ITSI Service dependencies, and Service Templates which include Request Rate, Error Rate, and Duration (latency) are applied for the Business Workflow and associated APM Services.

What is a Splunk APM Business Workflow?

In Splunk APM, a Business Workflow is the start-to-finish journey of the collection of traces associated with a given activity or transaction. It enables monitoring of end-to-end KPIs and identifying root causes and bottlenecks.

Business and engineering executives might use Business Workflows in Splunk APM to:

  • Monitor end-to-end rate, error, and duration (RED) metrics for most critical transactions
  • Ensure the organization is meeting associated KPIs, such as What is the 90th percentile of the logins duration? or What percentage of checkout requests return an error?

Site reliability engineers and service owners might use Business Workflows in Splunk APM to:

  • Troubleshoot issues
  • Narrow down context to only those services involved in a transaction of interest. For example: Which downstream service is responsible for increased response of the API class? or Are the errors in the Login service only happening for API calls from the homepage?

For more information, configuration instructions, and examples of Business Workflows, see Correlate traces to track Business Workflows.

Prerequisites

To use the Import Splunk APM Business Workflows functionality, the Splunk Infrastructure Monitoring Add-on must be installed and configured.

Ensure that the following Saved Searches are enabled:

  • ITSI Import Objects - Splunk APM Business Workflows
  • ITSI Import Objects - Splunk APM Business Workflow Entities

Import Splunk APM Workflows

To import configured Business Workflows, navigate to the Splunk Observability - Import Business Workflows Dashboard.

Keep the following tips in mind while using the Business Workflows configuration dashboard:

  • Avoid changing the name of any service that is imported from the "Business Workflows" Dashboard.
  • When publishing a sandbox generated from the "Business Workflows" dashboard, do not enable the backfill option because such sandboxes cannot be reverted.

Note: Service sandbox functionality does not allow for editing existing services in ITSI.

Perform the following procedure when importing any service tree into sandboxes using the "Business Workflows" dashboard:

  1. Create the service named "Splunk APM Business Workflows" in ITSI. (If not created previously.)
  2. Import the required services trees into sandbox and open the sandbox.
  3. Delete "Splunk APM Business Workflows" service from the sandbox.
  4. Make a note of all the parent (top-level) nodes and publish the services into ITSI from the sandbox.
  5. Manually add the parent (top-level) nodes as "Service Dependencies" of "Splunk APM Business Workflows" Service.

To view the Import Business Workflows configuration dashboard, follow these steps:

  1. Select Dashboards > Dashboards from the ITSI menu.
  2. Enter "Splunk Observability - Import Business Workflows" in the filter field to filter the list of dashboards.
  3. Select the dashboard to view.

border alt="Screenshot of the Business Workflows configuration dashboard

On the configuration page, select Import APM Business Workflow to open a guided configuration modal.

Step 1: Select the Splunk Observability Cloud Organization

The Splunk Observability Cloud Organizations which have been configured in the Splunk Infrastructure Monitoring Add-on are available in the configuration modal. Select the organization which includes the Business Workflows that you want to import.

border alt="Screenshot of Business Workflows modal step 1

Step 2: Select the Splunk Observability Cloud Environment

The Splunk Observability Cloud Environment(s) that are available in the organization you selected in the previous step are listed in the configuration modal. Select the Splunk Observability Cloud Environment(s) which include the Business Workflows you want to import.

border alt="Screenshot of Business Workflows modal step 2

Step 3: Select the Splunk Observability Cloud APM Business Workflows

Select the Business Workflow(s) that you want to import. If multiple environments were selected in the previous step, the Business Workflows will be grouped by environment in the dropdown.

Note: If you want to import the entire Splunk APM topology for the selected environment(s) you can enable the '''Import Entire APM Topology''' toggle. When using this functionality, be sure to review the number of ITSI services that will be created depending on the complexity of your application and services instrumented with Splunk APM.

border alt="Screenshot of Business Workflows modal step 3

Step 4: Review selected Business Workflows

A preview of the ITSI Services that will be created for each Environment and Business Workflow selected in the previous steps is available for review.

Note: Placeholder services will be created for Splunk APM Inferred Services and Databases. These can be used along with custom ITSI KPIs to add additional context to your imported Business workflows.

border alt="Screenshot of Business Workflows modal step 4

Step 5: Import Services to ITSI

You can choose how you want to import the services into ITSI. Import the services to a Service Sandbox if you are using a version of ITSI that includes Service Sandbox functionality.

Note: Placeholder services are created for Splunk APM Inferred Services and Databases. These can be used along with custom ITSI KPIs to add additional context to your imported Business workflows.

border alt="Screenshot of Business Workflows modal step 5

Services can be imported directly into ITSI by disabling the Import Business Workflows to ITSI Service Sandbox toggle. When importing the services directly into ITSI, you can choose whether to enable the services using the Enable ITSI Services toggle.

border alt="Screenshot of Business Workflows modal step 6

Use imported Splunk APM Workflows

Once you have completed the Import Splunk APM Business Workflow configuration, ITSI services are created based on the topology of the Business Workflow or APM Topology.

The created ITSI services are namespaced using the Splunk Observability Cloud Environment name of the associated Business Workflow.

If you imported the Business Workflow(s) into a Service Sandbox, you can access the Service Sandbox in the configuration section of the Splunk Observability - Import Business Workflows dashboard or Configuration > Service Monitoring > Service Sandboxes in ITSI.

Note: .

You can make any customizations or modifications to the services, dependencies, KPIs, etc. in the Service Sandbox before publishing the services to ITSI.

border alt="Example of imported Business Workflow display

If the Business Workflow(s) were imported directly into ITSI, they will be available in the ITSI Service Analyzer.

Note: The Business Workflow ITSI services need to be enabled if they were not enabled during the import process.

Service, KPI, and Entity Details

The Content Pack for Splunk Observability Cloud ships with two preconfigured Service Templates, Splunk APM Business Workflow KPIs and Splunk APM Service KPIs, which are automatically linked with the imported Business Workflow services. These templates include relevant KPIs for Business Workflows and Splunk APM services.

For more information on the Service Templates and KPIs, see KPI Reference for the Content Pack for Splunk Observability Cloud.

KPI Base Searches

The Content Pack for Splunk Observability Cloud ships with three KPI base searches used for monitoring Business Workflows and Splunk APM Services:

  • DA-ITSI-APM:BusinessWorkflows
  • DA-ITSI-APM:ServiceEndpointKPIs
  • DA-ITSI-APM:ServiceKPIs

The KPIs are automatically associated with the appropriate ITSI Services using the Service Templates outlined above. By default, the KPI Base Searches are scheduled to run every 5 minutes.

Entities

The SplunkAPM Entity Type is used for Splunk APM Services that are imported. Entity drilldowns are available for the following Splunk APM Service KPIS:

  • Request Rate
  • Error Rate
  • Request Latency (median)
  • Request Latency (p90)
  • Request Latency (p99)

Pseudo-entities for Splunk APM Service Endpoints are created for the following KPIs:

  • Request Latency by Endpoint (p90)
  • Error Rate by Endpoint

For more information on the Entity Searches available in the Content Pack for Splunk Observability Cloud see Entity search reference for the Content Pack for Splunk Observability Cloud.

Saved Searches

Two saved searches are used during the Business Workflow import process:

  • ITSI Import Objects - Splunk APM Business Workflows
  • ITSI Import Objects - Splunk APM Business Workflow Entities

These searches are used to create the ITSI services using the Configuration dashboard and only executed during the import process. Accordingly, they are not configured to run on a schedule.

Note: These searches require Business Workflow topology data from Splunk Observability Cloud which is collected during the import process. They should not be configured to run on a schedule.