Analyze error spans in Splunk APM

How Splunk APM detects error spans

Each span in Splunk APM captures a single operation. Splunk APM considers a span to be an error span if the operation that the span captures results in an error as defined by the following conditions:

• The otel.status_code field for the span is ERROR. otel.status_code is set in the Splunk Distribution of the OpenTelemetry instrumentation using the native OTel field span.status. span.status, and subsequently otel.status_code, are set based on either the HTTP status code or the gRPC status code.

  • See How OpenTelemetry handles HTTP status codes to learn which status code values set otel.status_code to ERROR in the OpenTelemetry instrumentation.

  • See How OpenTelemetry handles gRPC status codes to learn which rpc.grpc.status_code tag values set otel.status_code to ERROR in the OpenTelemetry instrumentation.

• The error tag for the span is set to a truthy value, which is any value other than False or 0.

See the Span Status section of the OpenTelemetry Transformation to non-OTLP Formats spec on GitHub https://opentelemetry.io/docs/specs/otel/common/mapping-to-non-otlp/#span-status to learn more about otel.status_code. See the Set Status section of the OpenTelemetry Tracing API specification on GitHub https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status to learn more about span.status.

How OpenTelemetry handles HTTP status codes

The following table provides an overview of how HTTP status codes are used to set the span.status field, and subsequently otel.status_code, in OpenTelemetry instrumentation in accordance with OpenTelemetry semantic conventions. To learn more, see the OpenTelemetry semantic conventions for HTTP spans on GitHub https://github.com/open-telemetry/semantic-conventions/blob/main/model/http/spans.yaml.

How OpenTelemetry handles gRPC status codes

To determine if a gRPC span counts towards the error rate for a service, Splunk APM looks at the otel.status_code field as set by OpenTelemetry instrumentation. The following logic is applied by the instrumentation in accordance with OpenTelemetry semantic conventions:

See the OpenTelemetry specification for information on the handling of gRPC status codes on GitHub https://github.com/open-telemetry/semantic-conventions/blob/main/model/rpc/spans.yaml.

How error spans are counted in MetricSets

To generate endpoint-level Monitoring MetricSets, Splunk APM turns endpoint spans, which are spans with span.kind = SERVER or span.kind = CONSUMER, into error metric data. If a span is considered an error per the Error rules in Splunk APM, that span counts towards errors in the Monitoring MetricSet for the endpoint associated with that span.

Service-level Monitoring MetricSets are based on the number of error spans in each of the service endpoints.

Server-side and client-side error counting

Splunk APM captures all spans from all instrumented services, including spans capturing requests made to clients, called client-side spans, and requests received by services, called server-side spans. In certain cases, when a service returns an error, the error can be registered in both the initiating span and the receiving span. To avoid duplicated error reports, Splunk APM counts only the server-side error spans in MetricSets and error totals.

For example, when service_a makes a call to service_b and both services are fully instrumented, Splunk APM receives the following two spans:

• span_1, a span with span.kind = CLIENT that captures service_a making the call to service_b

• span_2, a span with span.kind = SERVER that captures service_b receiving the request

If service_b returns a 500 error, both spans receive that error. To avoid double-counting errors, Splunk APM counts only the server-side span, span_2, as an error in MetricSets and error totals.

What is the difference between an error and a root cause error?

To help you identify the root cause of an error, Splunk APM differentiates between errors and root cause errors. For example, the request and error graph in Tag Spotlight differentiates root cause errors from total errors with a darker color:

When a particular span within a trace results in an error, the error can propagate through other spans in the trace. Any span determined to contain an error based on the criteria described in How Splunk APM detects error spans is an error span. Splunk APM designates the originating error of a chain of error spans as the root cause error.

For example, consider the checkout trace in the following screenshot:

The checkout service makes HTTP requests to the authorization service, the checkout service, and the payment service. The HTTP request to the payment service results in a 402 "Payment Required" error. Because the request to the payment service failed, the initiating requests to checkout service and http.Request also result in errors.

In this case, the source error, or root cause error, is the 402 error in the payment service. The 500 errors appearing in the checkout and api services are subsequent errors.

The root cause error count indicates the count of these root cause errors, while the standard error count indicates the total count of all root cause errors as well as any subsequent errors.

Customize the error logic in Splunk APM

To define which HTTP error codes contribute to the error rate metric in APM, you can update the default configuration for all services or configure a custom override for a specific service. Custom overrides take priority over the default configuration.

Error code metric configurations are applied at the organization level, as services may belong to more than one environment.

Update the default configuration for the error rate metric

1. In the Splunk Observability Cloud main menu, select Settings > APM error code configuration.

2. In the Default configuration table, select the edit icon under Actions.

3. Follow the on-screen steps to update the default configuration for all services.

Configure a custom override for the error rate metric

1. In the Splunk Observability Cloud main menu, select Settings > APM error code configuration.

2. Select Configure custom override.

3. Follow the on-screen steps to configure a custom override for a service.