Configure the Java agent 1.x for Splunk Observability Cloud (Deprecated)

Version 1.x of the Splunk Java Agent is deprecated. Learn how to migrate to the latest version.

The following sections describe all available settings for configuring the Java agent 1.x, including options for activating new features that are unique to the Splunk Distribution of OpenTelemetry Java.

Configuration methods

You can change the agent settings in two ways:

Set an environment variable. For example:

Linux

export OTEL_SERVICE_NAME=my-java-app

Windows PowerShell

$env:OTEL_SERVICE_NAME=my-java-app

Add a system property as runtime parameter. For example:

java -javaagent:./splunk-otel-javaagent.jar \
-Dotel.service.name=<my-java-app> \
-jar <myapp>.jar

Environment variables are the preferred way of configuring OpenTelemetry agents. System properties, if specified, override existing environment variables.

General settings

The following settings are specific to the Splunk Distribution of OpenTelemetry Java:


Environment variable	Description
`SPLUNK_REALM`	The name of your organization’s realm, for example, `us0`. When you set the realm, telemetry is sent directly to the ingest endpoint of Splunk Observability Cloud, bypassing the Splunk Distribution of OpenTelemetry Collector. System property: `splunk.realm`
`SPLUNK_ACCESS_TOKEN`	A Splunk authentication token that lets exporters send data directly to Splunk Observability Cloud. Unset by default. Not required unless you need to send data to the Splunk Observability Cloud ingest endpoint. See Create and manage authentication tokens using Splunk Observability Cloud. System property: `splunk.access.token`
`SPLUNK_TRACE_RESPONSE_HEADER_ENABLED`	Activates the addition of server trace information to HTTP response headers. For more information, see Server trace information. The default value is `true`. System property: `splunk.trace-response-header.enabled`
`SPLUNK_METRICS_FORCE_FULL_COMMANDLINE`	System property: `splunk.metrics.force_full_commandline`

Trace configuration

The following settings control tracing limits and attributes:


Environment variable	Description
`OTEL_SERVICE_NAME`	Name of the service or application you’re instrumenting. Takes precedence over the service name defined in the `OTEL_RESOURCE_ATTRIBUTES` variable. System property: `otel.service.name`
`OTEL_RESOURCE_ATTRIBUTES`	Comma-separated list of resource attributes added to every reported span. For example, `key1=val1,key2=val2`. System property: `otel.resource.attributes`
`OTEL_INSTRUMENTATION_COMMON_` `PEER_SERVICE_MAPPING`	Used to add a `peer.service` attribute by specifying a comma-separated list of mapping from hostnames or IP addresses. For example, if set to `1.2.3.4=cats-srv,dogs-srv.com=dogs-api`, requests to `1.2.3.4` have a `peer.service` attribute of `cats-service` and requests to `dogs-srv.com` have one of `dogs-api`. System property: `otel.instrumentation.common` `.peer-service-mapping`
`OTEL_INSTRUMENTATION_METHODS_INCLUDE`	Adds `@WithSpan` annotation functionality for the target method string. Format is `my.package.MyClass1[m1,m2];my.package.MyClass2[m3]`. System property: `otel.instrumentation.methods.include`
`OTEL_INSTRUMENTATION_OPENTELEMETRY` `_ANNOTATIONS_EXCLUDE_METHODS`	Suppresses `@WithSpan` instrumentation for specific methods. Format is `my.package.MyClass1[m1,m2];my.package.MyClass2[m3]`. System property: `otel.instrumentation.opentelemetry` `-annotations.exclude-methods`
`OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT`	System property: `otel.span.attribute.count.limit`
`OTEL_SPAN_EVENT_COUNT_LIMIT`	System property: `otel.span.event.count.limit`
`OTEL_SPAN_LINK_COUNT_LIMIT`	Maximum number of links per span. The default value is `1000`. System property: `otel.span.link.count.limit`

Exporters configuration

The following settings control trace exporters and their endpoints:


Environment variable	Description
`OTEL_TRACES_EXPORTER`	System property: `otel.traces.exporter`
`OTEL_EXPORTER_OTLP_ENDPOINT`	OTLP gRPC endpoint. The default value is `http://localhost:4317`. System property: `otel.exporter.otlp.endpoint`

The Splunk Distribution of OpenTelemetry Java uses the OTLP gRPC span exporter by default. To send data directly to Splunk Observability Cloud, see Send data directly to Splunk Observability Cloud.

Samplers configuration

The following settings control trace sampling:


Environment variable	Description
`OTEL_TRACES_SAMPLER`	Sampler to use. The default value is `always_on`. In addition to the samplers provided by the OpenTelemetry Java SDK, you can use the following samplers: `internal_root_off`: Drops all traces with root spans where `spanKind` is `INTERNAL`, `CLIENT` or `PRODUCER`. Keeps root spans where `spanKind` is `SERVER` or `CONSUMER`. `rules`: Drops all traces that originate from specific endpoints, as defined by the value of the `OTEL_TRACES_SAMPLER_ARG` setting. Only applies to spans where `spanKind` is `SERVER`. System property: `otel.traces.sampler`
`OTEL_TRACES_SAMPLER_ARG`	Semicolon-separated list of rules for the `rules` sampler. For example: `OTEL_TRACES_SAMPLER_ARG=drop=/healthcheck;fallback=always_on` The following rules are supported: `drop=<value>`: The sampler drops a span if its `http.target` attribute has a substring equal to the value you’ve provided. For example: `drop=/status`. `fallback=<sampler>`: Sampler to use if no `drop` rule matched a given span. Supported samplers are `always_on` and `parentbased_always_on`. If you define multiple fallback samplers, the Java agent uses the last one. If you don’t set arguments when using the `rules` sampler, the instrumentation defaults to the `parentbased_always_on` sampler. System property: `otel.traces.sampler.arg`

Example of trace sampling

The following example shows how to use the rules traces sampler to exclude the /healthcheck endpoint from monitoring:

export OTEL_TRACES_SAMPLER=rules
export OTEL_TRACES_SAMPLER_ARG=drop=/healthcheck;fallback=parentbased_always_on

All requests to downstream services that happen as a consequence of calling an excluded endpoint are also excluded.

Propagators configuration

The following settings control trace propagation:


Environment variable	Description
`OTEL_PROPAGATORS`	Comma-separated list of propagators you want to use. The default value is `tracecontext,baggage`. You can find the list of supported propagators in the OpenTelemetry documentation. System property: `otel.propagators`

For backward compatibility with older versions of the Splunk Distribution of OpenTelemetry Java or the SignalFx Java Agent, use the b3multi trace propagator:

Linux

export OTEL_PROPAGATORS=b3multi

Windows PowerShell

$env:OTEL_PROPAGATORS=b3multi

Java settings for AlwaysOn Profiling

The following settings control the AlwaysOn Profiling feature for the Java agent:


Environment variable	Description
`SPLUNK_PROFILER_ENABLED`	Activates AlwaysOn Profiling. The default value is `false`. System property: `splunk.profiler.enabled`
`SPLUNK_PROFILER_LOGS_ENDPOINT`	The collector endpoint for profiler logs. By default, it takes the value of `otel.exporter.otlp.endpoint`. System property: `splunk.profiler.logs-endpoint`
`SPLUNK_PROFILER_DIRECTORY`	The location of the JDK Flight Recorder files. The default value is the local directory (`.`). System property: `splunk.profiler.directory`
`SPLUNK_PROFILER_RECORDING_DURATION`	The duration of the recording unit. You can define duration in the form `<number><unit>`, where the unit can be `ms`, `s`, `m`, `h`, or `d`. The default interval is `20s`. If you enter a number but not a unit, the default unit is assumed to be `ms`. System property: `splunk.profiler.recording.duration`
`SPLUNK_PROFILER_KEEP_FILES`	Whether to preserve JDK Flight Recorder (JFR) files or not. The default value is `false`, which means that JFR files are deleted after processing. System property: `splunk.profiler.keep-files`
`SPLUNK_PROFILER_CALL_STACK_INTERVAL`	System property: `splunk.profiler.call.stack.interval`
`SPLUNK_PROFILER_MEMORY_ENABLED`	Activates memory profiling with all the options. To activate or deactivate specific memory profiling options, set their values explicitly. The default value is `false`. Requires `splunk.profiler.enabled` to be set to `true`. Activating memory profiling sets the value of `splunk.metrics.enabled` to `true`. Note: OpenJDK versions 15.0 to 17.0.8, are not supported for memory profiling. See https://bugs.openjdk.org/browse/JDK-8309862 in the JDK bug system for more information. System property: `splunk.profiler.memory.enabled`
`SPLUNK_PROFILER_MEMORY_EVENT_RATE`	Rate limit for memory profiling data, expressed as stack traces per unit of time. You can define duration in the form `<number>/<unit>`, where the unit can be `s` or `m`. The default value is `150/s`, or 150 stack traces per second. Consider increasing this value when collecting memory profiling data from complex, multithreaded workloads, like application servers. System property: `splunk.profiler.memory.event.rate`
`SPLUNK_PROFILER_TLAB_ENABLED`	Whether to activate TLAB memory events. The default value is the value assigned to the `splunk.profiler.memory.enabled` property. System property: `splunk.profiler.tlab.enabled`
`SPLUNK_PROFILER_INCLUDE_INTERNAL_STACKS`	Whether to include stack traces of the agent internal threads and stack traces with JDK internal frames. The default value is `false`. System property: `splunk.profiler.include.internal.stacks`
`SPLUNK_PROFILER_TRACING_STACKS_ONLY`	Whether to include only stack traces that are linked to a span context. The default value is `false`. When set to `true`, call stacks not linked to span contexts are dropped, which is useful to reduce data ingest volume. System property: `splunk.profiler.tracing.stacks.only`

For more information on AlwaysOn Profiling, see Introduction to AlwaysOn Profiling for Splunk APM.

Metrics collection settings

Note: To migrate from OpenTelemetry native metrics, seeMigration guide for OpenTelemetry Java 2.x metrics.

The following settings control metrics collection for the Java agent:


Environment variable	Description
`SPLUNK_METRICS_ENABLED`	Activates exporting metrics. See Metrics and attributes collected by the Splunk OTel Java agent for more information. Default is `false`. If you activate memory profiling using the `splunk.profiler.memory.enabled` property, the value of `splunk.metrics.enabled` is set to `true`. System property: `splunk.metrics.enabled`
`SPLUNK_METRICS_ENDPOINT`	The OTel collector metrics endpoint. Default is `http://localhost:9943/v2/datapoint`. System property: `splunk.metrics.endpoint`
`SPLUNK_METRICS_EXPORT_INTERVAL`	Deprecated. Use `OTEL_METRIC_EXPORT_INTERVAL` instead. See Periodic exporting MetricReader in the official OpenTelemetry documentation.

Server trace information

To connect Real User Monitoring (RUM) requests from mobile and web applications with server trace data, trace response headers are activated by default. The instrumentation adds the following response headers to HTTP responses:

Access-Control-Expose-Headers: Server-Timing
Server-Timing: traceparent;desc="00-<serverTraceId>-<serverSpanId>-01"

The Server-Timing header contains the traceId and spanId parameters in traceparent format. For more information, see the Server-Timing and traceparent documentation on the W3C website.

The following server frameworks and libraries add Server-Timing information:

Servlet API versions 2.2 to 4.X.
Netty versions 3.8 to 4.0.

Note: If you need to deactivate trace response headers, set SPLUNK_TRACE_RESPONSE_HEADER_ENABLED to false.

Other settings


Environment variable	Description
`OTEL_JAVAAGENT_ENABLED`	Globally activates the Java agent automatic instrumentation. The default value is `true`. Useful for deactivating automatic discovery in testing scenarios or pipelines. System property: `otel.javaagent.enabled`

Documentation