Spans Not Being Processed

If some spans are not being processed by Splunk AppDynamics, try the following:

Verify the required attributes for outgoing calls

For spans instrumenting an outgoing call, Splunk AppDynamics requires unique attributes appended to the Span Object or else the Span will get dropped from the Splunk AppDynamics pipeline. See the tables below for both required and recommended attributes for spans.

Some attributes are not required but are still highly recommended in order to get a more accurate depiction in the Flow Map.

Table 1. HTTP
Attribute Type Description Example(s) Requirement
http.method string HTTP request method GET, POST, HEAD Required
http.url string Full HTTP request URL https://www.foo.bar/search?q=OpenTelemetry#SemConv *Conditional Requirement

http.host

 string value of HTTP host header www.example.org

*Conditional Requirement
http.target string The full request target as passed in a HTTP request line or equivalent. /path/12314/?q=ddds#123 *Conditional Requirement
http.scheme string Scheme identifying the used protocol http, https *Conditional Requirement
net.peer.ip string Remote address of the peer 127.0.0.1 *Conditional Requirement
net.peer.port int Remote port number 80, 8080, 443 *Conditional Requirement
net.peer.name string Remote hostname or similar example.com Required

* Conditional Requirement - One of the following must be included at span creation time:

  • http.url
  • http.scheme , http.host , http.target
  • http.scheme , net.peer.name , net.peer.port , http.target
  • http.scheme , net.peer.ip , net.peer.port , http.target
Table 2. Database
Attribute Type Description Example(s) Requirement
db.system string An identifier for the database management system (DBMS) product being used other_sql Required
net.peer.ip string Remote address of the peer (dotted decimal for IPv4 or RFC5952 for IPv6) 127.0.0.1 *Conditional Requirement
net.peer.name string Remote Hostname example.com *Conditional Requirement

* Conditional Requirement- You need to use either net.peer.ip or net.peer.name.

Table 3. Messaging Queues
Attribute Type Description Example(s) Requirement
messaging.system string A string identifying the Messaging System kafka, rabbitmq, rocketmq, activemq Required
messaging.destination string The message destination name. This might be equal to the span name but is required nevertheless. MyQueue, MyTopic Required

Change the default SpanKind

Splunk AppDynamics does not process the default span kind INTERNAL. If your implementation is sending INTERNAL spans, those spans will not be reflected in the Controller UI Flow Maps. You must set SpanKind to any of the following alternative span kinds:

  • SERVER - Indicates that the span covers server-side handling of a synchronous RPC or other remote request. This span is often the child of a remote CLIENT span that was expected to wait for a response.
  • CLIENT - Indicates that the span describes a request to some remote service. This span is usually the parent of a remote SERVER span and does not end until the response is received.
  • PRODUCER - Indicates that the span describes the initiators of an asynchronous request. This parent span will often end before the corresponding child CONSUMER span, possibly even before the child span starts. In messaging scenarios with batching, tracing individual messages requires a new PRODUCER span per message to be created.
  • CONSUMER - Indicates that the span describes a child of an asynchronous PRODUCER request.

SERVER and CONSUMER spans are entry spans, whereas CLIENT and PRODUCER spans are exit spans. An entry span must precede an exit span to make the exit calls discoverable. Also, one or more INTERNAL spans can be present between an entry and exit span as long as the entry span precedes the exit span.

See OpenTelemetry Specifications for more information.

Here is an example from OpenTelemetry documentation to set SpanKind to SERVER using the OpenTelemetry SDK (Java):

Span span = tracer.spanBuilder("/resource/path").setSpanKind(SpanKind.SERVER).startSpan();
span.setAttribute("http.method", "GET");
span.setAttribute("http.url", url.toString());