Implement a telemetry routing scenario

Reference information for selecting and implementing telemetry routing scenarios for .NET applications.

Overview

Decide on a telemetry route (in other words, how traces and metrics leave the application host or Kubernetes cluster and arrive in Splunk Observability Cloud).

Implement a telemetry routing scenario in a non-production environment first. Use dual mode for a short time so as to not impact your work in Splunk AppDynamics while you set up Splunk Observability Cloud. Use what you learn in your non-production environment to ensure that your production environment can also support dual mode and make adjustments to avoid an impact on your production application.

Scenario Use this scenario when Route-specific agent settings

Send traces and metrics to a local OpenTelemetry Collector

You run a Collector on the same host or at a host-local endpoint and want the recommended production route for exporting telemetry to Splunk Observability Cloud. This route provides the recommended routing.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 or another local Collector OTLP/HTTP endpoint.

Send traces and metrics directly to Splunk Observability Cloud

You do not need a Collector for this application, your scenario supports direct ingest, and the route suits a small deployment or early testing. Direct ingest in production environments requires an exception and approval from Splunk.

SPLUNK_REALM=<realm> and SPLUNK_ACCESS_TOKEN=<token>.

Send traces and metrics to an OpenTelemetry gateway

You centralize telemetry through a gateway Collector to reduce direct connections from application environments to Splunk Observability Cloud.

OTEL_EXPORTER_OTLP_ENDPOINT=http://<gateway-host>:4318, unless the confirmed gateway endpoint differs.

Send traces and metrics directly to Splunk Observability Cloud in Kubernetes

The application runs in Kubernetes, and the Kubernetes workload uses an approved direct-ingest route without a local Collector agent.

Set SPLUNK_REALM and SPLUNK_ACCESS_TOKEN as workload environment variables, preferably from Kubernetes secrets.

Send traces and metrics to the local OpenTelemetry agent in Kubernetes

The application runs in Kubernetes, a local OpenTelemetry Collector agent runs in the cluster and can receive telemetry from the workload.

Set OTEL_EXPORTER_OTLP_ENDPOINT to the local Collector agent endpoint for the pod or node.

Scenario 1: send traces and metrics to a local OpenTelemetry Collector

The application sends traces and metrics to the local Collector. The Collector exports telemetry to Splunk Observability Cloud.

Confirm that the application process can reach the local Collector OTLP/HTTP endpoint.

  1. Configure the required .NET combined agent settings.

    For example:

    CODE 
    OTEL_SERVICE_NAME=<tier-name>
OTEL_RESOURCE_ATTRIBUTES=service.namespace=<application-name>,deployment.environment.name=<account-name>
  2. For this scenario, confirm that the environment variable (or equivalent setting) OTEL_EXPORTER_OTLP_ENDPOINT points to the local Collector OTLP/HTTP endpoint, typically http://localhost:4318.
  3. Restart the application or host process.

Failure Symptoms and Recovery Path

Failure symptom Recovery path

No service appears in Splunk Observability Cloud.

Confirm that the application process picked up the combined agent settings and that the Collector runs on the configured endpoint.

Traces appear but metrics do not appear, or metrics appear but traces do not appear.

Confirm OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, Collector pipelines, and Collector exporters.

The Collector logs authentication, endpoint, queue, or export errors.

Confirm the Splunk Observability Cloud realm, access token, network access, and Collector exporter configuration.

Escalation Evidence to Collect

  • .NET combined agent version, .NET runtime or .NET Framework version, operating system, architecture, and hosting model.
  • Redacted combined-agent environment variables or configuration files.
  • Redacted Collector configuration.
  • Splunk AppDynamics application, tier, and node names.
  • Splunk Observability Cloud realm, service name, service namespace, and deployment environment.
  • Agent logs, Collector logs, timestamps, example trace IDs, and metric names.

Scenario 2: send traces and metrics directly to Splunk Observability Cloud

The application sends traces and metrics directly to Splunk Observability Cloud. In dual-signal mode, the application also continues reporting to Splunk AppDynamics.

Confirm that:

  • The application and migration wave have direct-ingest approval.
  • The application host can reach Splunk Observability Cloud ingest endpoints.
  1. Configure the required .NET combined agent settings.

    For example:

    CODE 
    OTEL_SERVICE_NAME=<tier-name>
OTEL_RESOURCE_ATTRIBUTES=service.namespace=<application-name>,deployment.environment.name=<account-name>
  2. For this scenario, configure these environment variables (or equivalent settings):

    1. Set SPLUNK_REALM=<realm>.

    2. Set SPLUNK_ACCESS_TOKEN=<token> by using a secure configuration source.

    3. Do not set OTEL_EXPORTER_OTLP_ENDPOINT unless you intentionally want to override SPLUNK_REALM.

  3. Restart the application or host process.

Failure Symptoms and Recovery Path

Failure symptom Recovery path

No telemetry appears in Splunk Observability Cloud.

Confirm the realm, token, outbound network access, and whether OTEL_EXPORTER_OTLP_ENDPOINT overrides SPLUNK_REALM.

Logs or profiling data do not appear.

Use the Collector route. Direct ingest through SPLUNK_REALM supports traces and metrics only.

Telemetry appears under an unexpected service name or environment.

Confirm OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES.

Escalation Evidence to Collect

  • .NET combined agent version and supported-environment details.
  • Redacted agent configuration.
  • Splunk Observability Cloud realm and ingest path.
  • Agent logs, proxy details, firewall details, timestamps, example trace IDs, and metric names.

Scenario 3: send traces and metrics to an OpenTelemetry gateway

The application sends traces and metrics to the OpenTelemetry gateway. The gateway exports telemetry to Splunk Observability Cloud. In dual-signal mode, the application also continues reporting to Splunk AppDynamics.

Confirm that:

  • The application host can reach the gateway OTLP/HTTP endpoint.
  • The gateway can export traces and metrics to Splunk Observability Cloud.
  1. Configure the required .NET combined agent settings.

    For example:

    CODE 
    OTEL_SERVICE_NAME=<tier-name>
OTEL_RESOURCE_ATTRIBUTES=service.namespace=<application-name>,deployment.environment.name=<account-name>
  2. For this scenario, configure these environment variables (or equivalent settings):

    1. Set OTEL_EXPORTER_OTLP_ENDPOINT=http://<gateway-host>:4318, or the confirmed gateway OTLP/HTTP endpoint.

    2. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf when you need to make the protocol explicit.

  3. Restart the application or host process.

Failure Symptoms and Recovery Path

Failure symptom Recovery path

The gateway does not receive telemetry.

Confirm network access from the application host to the gateway and confirm the endpoint, port, and protocol.

The gateway receives telemetry but does not export it to Splunk Observability Cloud.

Confirm gateway exporters, access token, realm, and network access to Splunk Observability Cloud.

Traces or metrics appear under unexpected names.

Confirm service identity and resource attributes on the application host and in the gateway processors.

Escalation Evidence to Collect

  • .NET combined agent version and supported-environment details.
  • Redacted agent configuration and gateway configuration.
  • Gateway logs, agent logs, endpoint details, timestamps, example trace IDs, and metric names.

Scenario 4: send traces and metrics directly to Splunk Observability Cloud in Kubernetes

The Kubernetes workload sends traces and metrics directly to Splunk Observability Cloud. In dual-signal mode, the workload also continues reporting to Splunk AppDynamics.

Confirm that:

  • You want to keep the Splunk AppDynamics Cluster Agent path and do not want to install the OpenTelemetry Collector agent in the cluster.
  • The workload can reach Splunk Observability Cloud ingest endpoints.
  • Use Kubernetes secrets or an approved secret-management path to provide the Splunk Observability Cloud access token.
  1. Configure the required .NET combined agent settings.

    For example:

    CODE 
    OTEL_SERVICE_NAME=<tier-name>
OTEL_RESOURCE_ATTRIBUTES=service.namespace=<application-name>,deployment.environment.name=<account-name>
  2. For this scenario, configure these environment variables (or equivalent settings):

    1. Set SPLUNK_REALM=<realm>.

    2. Set SPLUNK_ACCESS_TOKEN from a Kubernetes secret.

  3. Apply the workload changes.
  4. Restart or roll out the workload.
  5. Verify that pods restart with the expected environment variables.

Failure Symptoms and Recovery Path

Failure symptom Recovery path

Pods start without instrumentation.

Confirm the workload manifest, agent package, environment variables, and mounted configuration.

No telemetry appears in Splunk Observability Cloud.

Confirm the realm, token secret, outbound network access, and whether the workload picked up the environment variables.

Telemetry appears without expected Kubernetes metadata.

Confirm the metadata requirements for this no-Collector scenario. If you require Kubernetes metadata enrichment, use a local Collector agent or another supported enrichment path.

Escalation Evidence to Collect

  • .NET combined agent version and container image details.
  • Workload manifest with secrets redacted.
  • Pod logs, agent logs, namespace, pod names, node names, timestamps, example trace IDs, and metric names.
  • Splunk AppDynamics Cluster Agent configuration when applicable.

Scenario 5: send traces and metrics to the local OpenTelemetry agent in Kubernetes

The Kubernetes workload sends traces and metrics to the local OpenTelemetry Collector agent. The Collector exports telemetry to Splunk Observability Cloud or to a gateway that exports to Splunk Observability Cloud. In dual-signal mode, the workload also continues reporting to Splunk AppDynamics.

  • The Kubernetes workload and container image support the .NET combined agent.
  • Deploy the Splunk Distribution of the OpenTelemetry Collector in the cluster, such as the Collector agent DaemonSet from the Helm chart.
  • The workload can reach the local Collector agent endpoint.
  • Configure the Collector agent to export traces and metrics to Splunk Observability Cloud or to a gateway.
  1. Configure the required .NET combined agent settings.

    For example:

    CODE 
    OTEL_SERVICE_NAME=<tier-name>
OTEL_RESOURCE_ATTRIBUTES=service.namespace=<application-name>,deployment.environment.name=<account-name>
  2. For this scenario, configure these environment variables (or equivalent settings):

    1. Set OTEL_EXPORTER_OTLP_ENDPOINT to the local Collector agent OTLP/HTTP endpoint for the workload.

    2. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf when you need to make the protocol explicit.

  3. Apply the workload changes.
  4. Restart or roll out the workload.
  5. Verify that pods restart with the expected environment variables.

Failure Symptoms and Recovery Path

Failure symptom Recovery path

The workload cannot reach the local Collector agent endpoint.

Confirm the endpoint, Kubernetes service, pod-to-node routing, namespace policy, and Collector receiver configuration.

The Collector receives telemetry but does not export it.

Confirm the Collector exporter, access token, realm, gateway route, and network access.

Telemetry appears without expected Kubernetes metadata.

Confirm the Collector Kubernetes metadata processors and Helm values for the selected deployment mode.

Escalation Evidence to Collect

  • .NET combined agent version and container image details.
  • Workload manifest with secrets redacted.
  • Collector Helm values or Collector configuration with secrets redacted.
  • Pod logs, Collector logs, namespace, pod names, node names, timestamps, example trace IDs, and metric names.