Zero-code instrumentation for back-end applications in Linux

Get started with zero-code instrumentation for back-end applications in Linux environments.

The Splunk Distribution of the OpenTelemetry Collector uses automatic discovery with zero-code instrumentation to automatically detect back-end applications running in your Linux environment. By deploying the Collector with zero-code instrumentation, you can monitor applications and send data to Splunk Observability Cloud without editing your application’s code or configuring files.

Zero-code instrumentation for Linux can detect and configure the following applications and language runtimes:

  • Java

  • Node.js

  • .NET

How zero-code instrumentation for Linux works

Zero-code instrumentation for Linux operates as a mode of the Splunk Distribution of the OpenTelemetry Collector. You install and activate zero-code instrumentation for the Collector by using the Linux installer script or package manager. During installation, you can specify the types of language runtimes you want the Collector to detect.

After installation, the Collector runs in your Linux environment and listens for requests to your applications. When the Collector detects activity, it gathers telemetry data from your application runtime and sends this data to Splunk Application Performance Monitoring (APM).

Requirements

You need the following components to use zero-code instrumentation for back-end Linux applications:

  • systemd

  • curl

  • sudo

Make sure you’ve also installed the components specific to your language runtime:

Java

Java version 8 or higher and supported libraries. See Java agent compatibility and requirements for more information.

Node.js

Node.js version 14 or higher and supported libraries. See Splunk OTel JS compatibility and requirements for more information.

.NET

.NET version 6.0 or higher and supported libraries. See OpenTelemetry .NET instrumentation compatibility and requirements for more information.

Zero-code instrumentation for .NET is only supported for x86_64/AMD64 architectures.

Get started

To install and use zero-code instrumentation for Linux, follow these steps:

  1. Install the package

  2. Ensure the collector service is running

  3. View results in Splunk APM

Install the package

Using the installer script, you can install and activate zero-code instrumentation for either all supported applications on the host via the system-wide method or only for applications running as systemd services.

Java
Installer script

Using the installer script, you can install the zero-code instrumentation package for Java and activate zero-code instrumentation for Java for either all supported Java applications on the host via the system-wide method or for only Java applications running as systemd services.

Note: By default, zero-code instrumentation is activated for all languages (Java, Node.js, and .NET) when using the installer script. To deactivate zero-code instrumentation for other languages, add the --without-instrumentation-sdk [language] option in the installer script command.
System-wide

Run the installer script with the --with-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/java.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

systemd

Run the installer script with the --with-systemd-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The systemd instrumentation automatically adds environment variables to /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

Deb/RPM

Run the installer script with the --with-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/java.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

Ansible

Run the installer script with the --with-systemd-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The systemd instrumentation automatically adds environment variables to /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

Chef
Note: You must first install the Splunk OpenTelemetry Collector using the Install the Collector for Linux with downloaded packages.

After installing the Collector, follow these steps to install the package using the Debian or RPM repositories with root privileges:

  1. You can either download the splunk-otel-auto-instrumentation package directly from the GitHub Releases page or add the Splunk repository to the package repositories on your Linux host. See Debian or RPM package repositories for instructions on how to configure your package repository.

  2. Run the following commands to install the package. Replace <path to splunk-otel-auto-instrumentation deb/rpm> with the local path to the downloaded package.

    Debian
    sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
    RPM
    sudo rpm -ivh <path to splunk-otel-auto-instrumentation rpm>

  3. Edit the /etc/otel/collector/splunk-otel-collector.conf file to set the SPLUNK_ACCESS_TOKEN and SPLUNK_REALM variables to the values you got earlier. If the file does not exist, use the provided sample at /etc/otel/collector/splunk-otel-collector.conf.example as a starting point.

    SPLUNK_ACCESS_TOKEN=<access_token>
SPLUNK_REALM=<realm>

  4. Start the collector service:

    sudo systemctl start splunk-otel-collector

  5. Ensure the collector service is running.

  6. Start your applications.

Puppet
sudo dpkg -i <path to splunk-otel-auto-instrumentation deb>
Salt
sudo rpm -ivh <path to splunk-otel-auto-instrumentation rpm>
Node.js

Using the installer script, you can install the zero-code instrumentation package for Java and activate zero-code instrumentation for Java for either all supported Java applications on the host via the system-wide method or for only Java applications running as systemd services.

Note: By default, zero-code instrumentation is activated for all languages (Java, Node.js, and .NET) when using the installer script. To deactivate zero-code instrumentation for other languages, add the --without-instrumentation-sdk [language] option in the installer script command.
System-wide

Run the installer script with the --with-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/java.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

systemd

Run the installer script with the --with-systemd-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The systemd instrumentation automatically adds environment variables to /usr/lib/systemd/system.conf.d/00-splunk-otel-auto-instrumentation.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-systemd-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

.NET

Run the installer script with the --with-instrumentation option, as shown in the following example. Replace <SPLUNK_REALM> and <SPLUNK_ACCESS_TOKEN> with your Splunk Observability Cloud realm and token, respectively.

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

The system-wide zero-code instrumentation method automatically adds environment variables to /etc/splunk/zeroconfig/java.conf.

To automatically define the optional deployment.environment resource attribute at installation time, run the installer script with the --deployment-environment <env> option. Replace <env> with the desired attribute value, for example, prod, as shown in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN>

You can activate AlwaysOn Profiling for CPU and memory, as well as metrics, using additional options, as in the following example:

curl -sSL https://dl.signalfx.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh && \
sudo sh /tmp/splunk-otel-collector.sh --with-instrumentation --deployment-environment prod \
--realm <SPLUNK_REALM> -- <SPLUNK_ACCESS_TOKEN> \
--enable-profiler --enable-profiler-memory --enable-metrics

Next, ensure the service is running and restart your application. See Ensure the collector service is running and Start your applications.

Ensure the Collector service is running

After a successful installation, run the following command to ensure the splunk-otel-collector service is running:

sudo systemctl status splunk-otel-collector

If the service is not running, start or restart it with the following command:

sudo systemctl restart splunk-otel-collector

If the service fails to start, check that the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN in /etc/otel/collector/splunk-otel-collector.conf are correct. You can also view the service logs with this command:

sudo journalctl -u splunk-otel-collector

Start your applications

For zero-code instrumentation to take effect, you must either restart the host or manually start or restart any applications on the host where you installed the package. You must restart the host or applications after installing the zero-code instrumentation package for the first time and whenever you make any changes to the configuration file.

After your applications are running, you can verify your data. See View results in Splunk APM. You can also configure instrumentation settings. See (Optional) Configure the instrumentation.

(Optional) Configure the instrumentation

You can configure the Splunk Distribution of the OpenTelemetry Collector to suit your instrumentation needs. In most cases, modifying the basic configuration is enough to get started.

To learn more, see the following resources:

Update zero-code instrumentation

Java

You can upgrade the package by using the package repository or by using Debian or RPM packages.

Package repository

If you installed the package using the installer script, or if you configured the Debian or RPM package repositories manually, run the following commands according to your platform. Upgrading the package requires root privileges.

Debian

Run the following commands:

sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation

You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.

RPM

For the RPM package management system, run the following commands:

yum:

sudo yum upgrade splunk-otel-auto-instrumentation

dnf:

sudo dnf upgrade splunk-otel-auto-instrumentation

zypper:

sudo zypper refresh
sudo zypper update splunk-otel-auto-instrumentation

After you’ve upgraded the packages, manually start or restart the Java applications on the host for the changes to take effect.

Debian/RPM packages

Run the following commands:

sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation

You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.

Node.js

If you installed the package using the installer script, or if you configured the Debian or RPM package repositories manually, run the following commands according to your platform. Upgrading the package requires root privileges.

Debian

Run the following commands:

sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation

You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.

RPM

For the RPM package management system, run the following commands:

yum:

sudo yum upgrade splunk-otel-auto-instrumentation

dnf:

sudo dnf upgrade splunk-otel-auto-instrumentation

zypper:

sudo zypper refresh
sudo zypper update splunk-otel-auto-instrumentation

After you’ve upgraded the packages, manually start or restart the Java applications on the host for the changes to take effect.

.NET

Run the following commands:

sudo apt-get update
sudo apt-get --only-upgrade splunk-otel-auto-instrumentation

You might see a prompt to keep or overwrite the configuration file(s) in /etc/splunk/zeroconfig/. If you choose to overwrite, the configuration file reverts to the default file provided by the upgraded package.

Troubleshooting

To troubleshoot common errors that occur when instrumenting applications, see the following troubleshooting guides:

View results in Splunk APM

After activating zero-code instrumentation, ensure your data is flowing into Splunk Observability Cloud. See Verify that your data is coming into Splunk APM.