Spans Not Being Processed

If some spans are not being processed by Splunk AppDynamics, try the following:

Verify the required attributes for outgoing calls
Change the default SpanKind

Verify the required attributes for outgoing calls

For spans instrumenting an outgoing call, Splunk AppDynamics requires unique attributes appended to the Span Object or else the Span will get dropped from the Splunk AppDynamics pipeline. See the tables below for both required and recommended attributes for spans.

Some attributes are not required but are still highly recommended in order to get a more accurate depiction in the Flow Map.

Table 1. HTTP
Attribute	Type	Description	Example(s)	Requirement
`http.method`	string	HTTP request method	GET, POST, HEAD	Required
`http.url`	string	Full HTTP request URL	https://www.foo.bar/search?q=OpenTelemetry#SemConv	*Conditional Requirement
http.host	string	value of HTTP host header	www.example.org	*Conditional Requirement
`http.target`	string	The full request target as passed in a HTTP request line or equivalent.	/path/12314/?q=ddds#123	*Conditional Requirement
`http.scheme`	string	Scheme identifying the used protocol	http, https	*Conditional Requirement
`net.peer.ip`	string	Remote address of the peer	127.0.0.1	*Conditional Requirement
`net.peer.port`	int	Remote port number	80, 8080, 443	*Conditional Requirement
`net.peer.name`	string	Remote hostname or similar	example.com	Required

* Conditional Requirement - One of the following must be included at span creation time:

http.url
http.scheme , http.host , http.target
http.scheme , net.peer.name , net.peer.port , http.target
http.scheme , net.peer.ip , net.peer.port , http.target

Table 2. Database
Attribute	Type	Description	Example(s)	Requirement
`db.system`	string	An identifier for the database management system (DBMS) product being used	other_sql	Required
`net.peer.ip`	string	Remote address of the peer (dotted decimal for IPv4 or RFC5952 for IPv6)	127.0.0.1	*Conditional Requirement
`net.peer.name`	string	Remote Hostname	example.com	*Conditional Requirement

* Conditional Requirement- You need to use either net.peer.ip or net.peer.name.

Table 3. Messaging Queues
Attribute	Type	Description	Example(s)	Requirement
`messaging.system`	string	A string identifying the Messaging System	kafka, rabbitmq, rocketmq, activemq	Required
`messaging.destination`	string	The message destination name. This might be equal to the span name but is required nevertheless.	MyQueue, MyTopic	Required

Change the default SpanKind

Splunk AppDynamics does not process the default span kind INTERNAL. If your implementation is sending INTERNAL spans, those spans will not be reflected in the Controller UI Flow Maps. You must set SpanKind to any of the following alternative span kinds:

SERVER - Indicates that the span covers server-side handling of a synchronous RPC or other remote request. This span is often the child of a remote CLIENT span that was expected to wait for a response.
CLIENT - Indicates that the span describes a request to some remote service. This span is usually the parent of a remote SERVER span and does not end until the response is received.
PRODUCER - Indicates that the span describes the initiators of an asynchronous request. This parent span will often end before the corresponding child CONSUMER span, possibly even before the child span starts. In messaging scenarios with batching, tracing individual messages requires a new PRODUCER span per message to be created.
CONSUMER - Indicates that the span describes a child of an asynchronous PRODUCER request.

SERVER and CONSUMER spans are entry spans, whereas CLIENT and PRODUCER spans are exit spans. An entry span must precede an exit span to make the exit calls discoverable. Also, one or more INTERNAL spans can be present between an entry and exit span as long as the entry span precedes the exit span.

See OpenTelemetry Specifications for more information.

Here is an example from OpenTelemetry documentation to set SpanKind to SERVER using the OpenTelemetry SDK (Java):

Span span = tracer.spanBuilder("/resource/path").setSpanKind(SpanKind.SERVER).startSpan();
span.setAttribute("http.method", "GET");
span.setAttribute("http.url", url.toString());