Get data into Call Graph Profiling

Learn how to get profiling data into Call Graph Profiling in Splunk APM.

Attention:

Alpha features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this alpha feature available in its sole discretion and may discontinue it at any time. These documents are not yet publicly available and we ask that you keep such information confidential. Use of alpha features is subject to the Splunk Pre-Release Agreement for Hosted Services.

Use the following links to navigate between the pages in this alpha release:

Complete the following high-level steps to get data into Call Graph Profiling.

Ensure that you meet the prerequisites. If you're deploying the Splunk Distribution of the OpenTelemetry Collector using Helm, you must complete an additional prerequisite.
Instrument your application or service.
Activate Call Graph Profiling.
- If you're using a gateway deployment, follow the instructions in Activate Call Graph Profiling in a gateway deployment.

Prerequisites

Learn about the prerequisites for getting data into Call Graph Profiling in Splunk APM.

To get data into Splunk APM Call Graph Profiling, you must meet the following requirements:

Splunk APM is activated for your Splunk Observability Cloud organization. For instructions, see Set up Splunk APM.
The Splunk Distribution of OpenTelemetry Collector version 0.44.0 or higher is running on your host. For instructions, see Get started with the Splunk Distribution of the OpenTelemetry Collector. If the version of your Collector is lower than 0.44.0, see Check the OpenTelemetry Collector configuration.
The Splunk Distribution of OpenTelemetry Java agent version 2.15.0 or higher is running on your application. For instructions, see Instrument your Java application for Splunk Observability Cloud. If the version of your Collector Java agent is lower than 2.15.0, see Upgrade the Splunk Distribution of OpenTelemetry Java.

Prerequisite for Helm chart deployments

Learn about the prerequisites for getting data into Call Graph Profiling if you are using a Helm chart deployment for the Splunk Distribution of the OpenTelemetry Collector.

If you're deploying the Splunk Distribution of the OpenTelemetry Collector using Helm, you must pass the following value when installing the chart:

--set splunkObservability.profilingEnabled='true'

You can also edit the parameter in the values.yaml file itself. For example:

# This option enables only the shared pipeline for logs and profiling data.
# There is no active collection of profiling data.
# Instrumentation libraries must be configured to send it to the collector.
# If you don't use Call Graph Profiling for Splunk APM, you can disable it.
profilingEnabled: false

Note: Setting profileEnabled to true creates the logs pipeline required by Call Graph Profiling, but doesn't install the APM instrumentation. To install the instrumentation, see the following section.

If you are using a version of the Collector lower than 0.78.0, turn off logs collection:

logsEnabled: false

Instrument your application or service

Learn how to instrument your application or service for Call Graph Profiling in Splunk APM.

Call Graph Profiling requires APM tracing data to correlate stack traces to your application requests. To instrument your application for Splunk APM, follow the steps for the appropriate programming language:


Language	Available instrumentation	Documentation
Java	Splunk Distribution of OpenTelemetry Java version 2.15.0 or higher	Instrument your Java application for Splunk Observability Cloud Java settings for Call Graph Profiling

In Java environments, Call Graph Profiling supports monitoring synchronous and asynchronous threads.

For information on data retention, see Data retention in Application Performance Monitoring (APM).

Activate Call Graph Profiling

Learn how to activate Call Graph Profiling in Splunk APM.

After you've instrumented your service for Splunk Observability Cloud and checked that APM data is getting into Splunk APM, you can activate Call Graph Profiling.

To activate Call Graph Profiling, check that the OTLP endpoint that exports profiling data is set correctly:

The profiling-specific endpoint is configured through the splunk.profiler.logs-endpoint system property or the SPLUNK_PROFILER_LOGS_ENDPOINT environment variable.
If that endpoint is not set, then the generic OTLP endpoint is used, configured through the otel.exporter.otlp.endpoint system property or the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.
If that endpoint is not set either, it defaults to http://localhost:4317.
For non-Kubernetes deployments, the OTLP endpoint has to point to http://${COLLECTOR_IP}:4317. If the collector and the profiled application run on the same host, then use http://localhost:4317. Otherwise, make sure there are no firewall rules blocking access to port 4317 from the profiled host to the collector host.
For Kubernetes deployments, the OTLP endpoint has to point to http://$(K8S_NODE_IP):4317 where the K8S_NODE_IP is fetched from the Kubernetes downstream API by setting the environment configuration on the Kubernetes pod running the application. For example:
```
env:
- name: K8S_NODE_IP
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: status.hostIP
```
Port 9943 is the default port for the SignalFx receiver in the collector distribution. If you change this port in your collector configuration, you need to pass the custom port to the JVM.

The following example shows how to activate the profiler using the system property:

java -javaagent:./splunk-otel-javaagent.jar \
-Dsplunk.snapshot.profiler.enabled=true \
-Dotel.exporter.otlp.endpoint=http(s)://collector:4317 \
-jar <your_application>.jar

For more configuration options, including setting a separate endpoint for profiling data, see Java settings for Call Graph Profiling.

Activate Call Graph Profiling in a gateway deployment

Learn how to activate Call Graph Profiling in a gateway deployment in Splunk APM.

Follow these steps to set up Call Graph Profiling with a collector in data forwarding or gateway mode, similar to the following example gateway deployment:

Step one. Point the instrumentation agent to the collector in host (agent) monitoring mode. Step two. Configure the collector in host (agent) monitoring mode. Step three. Configure the collector in data forwarding (gateway) mode. Step four. Send data to Splunk Observability Cloud.

Point the instrumentation agent to the OTLP gRPC receiver for the collector in host monitoring (agent) mode. The OTLP gRPC receiver must be running on the same host and port as the collector in host monitoring (agent) mode.
Configure the collector in host monitoring (agent) mode with the following components:
- An OTLP gRPC receiver
- An OTLP exporter pointed at the collector in data forwarding (gateway) mode
- A logs pipeline that connects the receiver and the exporter. For example, see the default agent configuration with the necessary adjustment to send to a gateway in the Splunk OpenTelemetry Collector on GitHub.
```
>>service:
   pipelines:
      logs:
         receivers: [otlp]
         processors:
         - memory_limiter
         - batch
         - resourcedetection
         #- resource/add_environment
         #exporters: [splunk_hec, splunk_hec/profiling]
         # Use instead when sending to gateway
         exporters: [otlp]
```
Configure the collector in data forwarding (gateway) mode with the following components:
1. An OTLP gRPC receiver.
2. A splunk_hec exporter.
3. A logs pipeline that connects the receiver and the exporter.

Next step

Learn what you can do after you get data into Call Graph Profiling.

After you get data into Call Graph Profiling, you can Analyze the performance of call stacks using call graphs.

Documentation