Generate customized instructions using the guided setup

To generate all the basic installation commands for your environment and application, use the Java guided setup. To access the Java guided setup, follow these steps:

1. Log in to Splunk Observability Cloud.

2. Open the Java guided setup. Optionally, you can navigate to the guided setup on your own:

   1. In the navigation menu, select Data Management.

   2. Go to the Available integrations tab, or select Add Integration in the Deployed integrations tab.

   3. In the integration filter menu, select By Product.

   4. Select the APM product.

   5. Select the Java tile to open the Java guided setup.

Install the Splunk Distribution of OpenTelemetry Java manually

If you don’t use the guided setup, follow these instructions to manually install the Splunk Distribution of OpenTelemetry Java:

• Install and activate the Java agent

  • Activate AlwaysOn Profiling

  • Metrics collection

  • Ignore specific endpoints

• Configure the Java agent

Install and activate the Java agent

Follow these steps to automatically instrument your application using the Java agent:

1. Check that you meet the requirements. See Java agent compatibility and requirements.

2. Download the JAR file for the latest version of the agent:

   Linux

   curl -L https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent.jar \
   -o splunk-otel-javaagent.jar

   Windows PowerShell

   Invoke-WebRequest -Uri https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent.jar -OutFile splunk-otel-javaagent.jar

3. Set the OTEL_SERVICE_NAME environment variable:

   Linux

   export OTEL_SERVICE_NAME=<yourServiceName>

   Windows PowerShell

   $env:OTEL_SERVICE_NAME=<yourServiceName>

4. (Optional) Set the endpoint URL if the Splunk Distribution of OpenTelemetry Collector is running on a different host:

   Linux

   export OTEL_EXPORTER_OTLP_ENDPOINT=<yourCollectorEndpoint>:<yourCollectorPort>

   Windows PowerShell

   $env:OTEL_EXPORTER_OTLP_ENDPOINT=<yourCollectorEndpoint>:<yourCollectorPort>

5. (Optional) Set the deployment environment and service version:

   Linux

   export OTEL_RESOURCE_ATTRIBUTES='deployment.environment=<envtype>,service.version=<version>'

   Windows PowerShell

   $env:OTEL_RESOURCE_ATTRIBUTES='deployment.environment=<envtype>,service.version=<version>'

6. Set the -javaagent argument to the path of the Java agent:

   java -javaagent:./splunk-otel-javaagent.jar -jar <myapp>.jar

   Note: If your application runs on a supported Java server, see Define agent paths for Java application servers for Splunk Observability Cloud.

If no data appears in APM, see Troubleshoot Java instrumentation for Splunk Observability Cloud.

If you need to add custom attributes to spans or want to manually generate spans, instrument your Java application or service manually. See Manually instrument Java applications for Splunk Observability Cloud.

Activate AlwaysOn Profiling

To activate AlwaysOn Profiling, use the following system property argument. You can also use the SPLUNK_PROFILER_ENABLED environment variable. For more information, see Introduction to AlwaysOn Profiling for Splunk APM.

To activate memory profiling, set the splunk.profiler.memory.enabled system property or the SPLUNK_PROFILER_MEMORY_ENABLED environment variable to true after activating AlwaysOn Profiling.

The following example shows how to activate the profiler using the system property:

java -javaagent:./splunk-otel-javaagent.jar \
-Dsplunk.profiler.enabled=true \
-Dsplunk.profiler.memory.enabled=true \
-Dotel.exporter.otlp.endpoint=http(s)://collector:4318 \
-Dsplunk.metrics.endpoint=http(s)://collector:4318
-jar <your_application>.jar

See Get data into Splunk APM AlwaysOn Profiling for more information. For more settings, see Java settings for AlwaysOn Profiling.

Metrics collection

Starting from version 2.5.0, the Java agent collects metrics by default when instrumenting an application or service automatically. To migrate metric collection from version 1.x to 2.x, see Migration guide for OpenTelemetry Java 2.x metrics.

If your metrics endpoint is different than the default value, set the OTEL_EXPORTER_OTLP_METRICS_ENDPOINT environment variable. See Configure the Java agent for Splunk Observability Cloud for more information.

If you activate memory profiling, metrics collection is activated automatically and cannot be deactivated.

Logs collection

By default, the Java agent injects trace and span metadata automatically into logs. The agent then sends the annotated logs to the OpenTelemetry Collector, which exports them to Splunk Observability Cloud. For more information on trace-log correlation, see Connect Java trace data with logs for Splunk Observability Cloud.

Ignore specific endpoints

By default, the Java agent collects traces from all the endpoints of your application. To ignore specific endpoints, use the rules sampler and define drop rules.

In the following example, the sampler drops all SERVER spans whose endpoints match healtcheck, and sends the rest of spans using the fallback sampler, parentbased_always_on:

Linux

export OTEL_TRACES_SAMPLER=rules
export OTEL_TRACES_SAMPLER_ARG=drop=/healthcheck;fallback=parentbased_always_on

Windows PowerShell

$env:OTEL_TRACES_SAMPLER=rules
$env:OTEL_TRACES_SAMPLER_ARG=drop=/healthcheck;fallback=parentbased_always_on

See Configure the Java agent for Splunk Observability Cloud for more information.

Configure the Java agent

You can configure the agent using environment variables or by setting system properties as runtime arguments. For more details about both methods, see Configuration methods.

For advanced configuration of the JVM agent, like changing trace propagation formats, correlating traces and logs, or activating custom sampling, see Configure the Java agent for Splunk Observability Cloud.

Deploy the Java agent in Kubernetes

To deploy the Java agent in Kubernetes, follow these steps:

1. Edit the Dockerfile for your application image to add the following commands:

   # Adds the latest version of the Splunk Java agent
   ADD https://github.com/signalfx/splunk-otel-java/releases/latest/download/splunk-otel-javaagent.jar .
   # Modifies the entry point
   ENTRYPOINT ["java","-javaagent:splunk-otel-javaagent.jar","-jar","./<myapp>.jar"]

2. Configure the Kubernetes Downward API to expose environment variables to Kubernetes resources.

   The following example shows how to update a deployment to expose environment variables by adding the agent configuration under the .spec.template.spec.containers.env section:

   apiVersion: apps/v1
   kind: Deployment
   spec:
   selector:
      matchLabels:
         app: your-application
   template:
      spec:
         containers:
         - name: myapp
            env:
               - name: SPLUNK_OTEL_AGENT
               valueFrom:
                  fieldRef:
                     fieldPath: status.hostIP
               - name: OTEL_EXPORTER_OTLP_ENDPOINT
               value: "http://$(SPLUNK_OTEL_AGENT):4318"
               - name: OTEL_SERVICE_NAME
               value: "<serviceName>"
               - name: OTEL_RESOURCE_ATTRIBUTES
               value: "deployment.environment=<environmentName>"

Deploy the Java agent in Cloudfoundry

To instrument a Java application in Cloudfoundry, use the Splunk OpenTelemetry Java Agent buildpack framework. The framework automatically instruments your application to send traces to Splunk Observability Cloud.

See https://github.com/cloudfoundry/java-buildpack/blob/main/docs/framework-splunk_otel_java_agent.md for instructions.

Send data directly to Splunk Observability Cloud

By default, the agent sends all telemetry to the local instance of the Splunk Distribution of OpenTelemetry Collector.

If you need to send data directly to Splunk Observability Cloud, set the following environment variables:

Linux

export SPLUNK_ACCESS_TOKEN=<access_token>
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.<realm>.signalfx.com/v2/trace/otlp
export OTEL_LOGS_EXPORTER=none

Windows PowerShell

$env:SPLUNK_ACCESS_TOKEN=<access_token>
$env:OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
$env:OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.<realm>.signalfx.com/v2/trace/otlp
$env:OTEL_LOGS_EXPORTER=none

To obtain an access token, see Retrieve and manage user API access tokens using Splunk Observability Cloud.

To find your Splunk realm, see Configure SSO integrations for Splunk Observability Cloud .

For more information on the ingest API endpoints, see Send APM traces.

Set the source host

Instrument Lambda functions

You can instrument AWS Lambda functions using the Splunk OpenTelemetry Lambda Layer. See Instrument your AWS Lambda function for Splunk Observability Cloud for more information.

Upgrade the Splunk Distribution of OpenTelemetry Java

New releases of the Splunk Distribution of OpenTelemetry Java happen after a new upstream release, or when new features and enhancements are available.

Upgrade to each new version of the Splunk Distribution of OpenTelemetry Java after it’s released. To find out about new releases, watch the GitHub repository at https://github.com/signalfx/splunk-otel-java/releases

Best practices for upgrades

To reduce the risk of issues with an upgrade, do the following:

• See the release notes and changelog for each release to determine if the release has changes that might affect your environment. Pay attention to mentions of libraries, frameworks, and tools that your software uses.

• Never put untested code into production. Verify that the new build works in a staging or pre-production environment before promoting it to production. Don’t use snapshot builds in production.

• Use canary instances. Let the canaries operate with the code before releasing the code to production. Run the canaries for at least a few hours, and preferably for a few days.

• Minimize the number of dependencies, including instrumentation, that change in a given release. Determining the root cause of a problem after upgrading multiple dependencies at the same time can be difficult.

• Pin version numbers in your build pipeline. Don’t use the latest URL in automated processes.