Upgrade the Collector for Kubernetes and other updates

Upgrade the Splunk Distribution of OpenTelemetry Collector for Kubernetes.

Upgrade the Collector for Kubernetes

The installer script uses one of the supported package managers to install the Collector. When you update the Collector using the official packages, configuration files are never overridden. If you need to update the configuration after an update, edit them manually before backward compatibility is dropped.

To upgrade the Collector for Kubernetes run the following commands:

  • Use the flag --reuse-values to keep the config values you’d already set while installing or using the Collector:

BASH
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector
--reuse-values
  • Use --values config.yaml to override your previous configuration while upgrading:

BASH
helm upgrade splunk-otel-collector --values config.yaml splunk-otel-collector-chart/splunk-otel-collector --reuse-values

Read more in the official Helm upgrade options documentation.

Upgrade guidelines

Apply the following changes to the Collector configuration files for specific version upgrades. For more details refer to Helm chart upgrade guidelines in GitHub.

From 0.113.0 to 0.116.0

Custom resource definition (CRD) configuration has been modified.

  • Before v0.110.0 CRDs were deployed via a crds/ directory (upstream default).

  • From v0.110.0 to v1.113.0 CRDs were deployed using Helm templates (upstream default), which had reported issues.

  • From v0.116.0 and higher, you must explicitly configure your preferred CRD deployment method or deploy the CRDs manually to avoid potential issues. You can deploy CRDs via a crds/ directory again by enabling a newly added value.

New users

If you’re a new user deploy CRDs via the crds/ directory. For a fresh installation use the following Helm values:

YAML
operatorcrds:
  install: true
operator:
  enabled: true

To install the chart run:

BASH
helm install <release-name> splunk-otel-collector-chart/splunk-otel-collector --set operatorcrds.install=true,operator. enabled=true <extra_args>

Current users

You might need to migrate if using operator.enabled=true.

If you’re using versions 0.110.0 to 1.113.0, CRDs are likely deployed via Helm templates. To migrate to the recommended crds/ directory deployment:

  1. Delete the existing chart running

    BASH
    helm delete <release-name>
  2. Verify if the following CRDs are present and delete them if necessary:

    BASH
    kubectl get crds | grep opentelemetry
    kubectl delete crd opentelemetrycollectors.opentelemetry.io
    kubectl delete crd opampbridges.opentelemetry.io
    kubectl delete crd instrumentations.opentelemetry.io
  3. Reinstall the chart with the updated configuration:

    BASH
    helm install <release-name> splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true <extra_args>

Current users maintaining legacy templates

If you’re using chart versions 0.110.0 to 1.113.0 and prefer to continue deploying CRDs via Helm templates (not recommended), use the following values:

YAML
operator:
  enabled: true
operator:
  crds:
    create: true
CAUTION: This method might cause race conditions during installation or upgrades.

From 0.105.5 to 0.108.0

Note: If you have no customizations under .Values.operator.instrumentation.spec.* no migration is required.

The Helm chart configuration for operator auto-instrumentation has been simplified, and the values previously under .Values.operator.instrumentation.spec.* have been moved to .Values.instrumentation.*.

The updated path looks like this:

YAML
instrumentation:
  endpoint: XXX
  ...

The deprecated path was:

YAML
operator:
  instrumentation:
    spec:
      endpoint: XXX
      ...

The installer script uses one of the supported package managers to install the Collector.

When you update the Collector using the official packages, configuration files are never overridden. If you need to update the configuration after an update, edit them manually before backward compatibility is dropped.

General guidelines

Apply the following changes to the Collector configuration files for specific version upgrades.

From 0.158.0 to 0.159.0

Linux DEB and RPM packages and Windows installations through MSI or Chocolatey now use otelcollauncher as the service entrypoint instead of otelcol. By default, the launcher starts otelcol directly and passes through the existing service arguments, so upgrades preserve the previous Collector service behavior. The launcher allows the service to start either otelcol directly or the OpAMP Supervisor, enabling additional Fleet Management capabilities. See OpenTelemetry Fleet Management for details.

For a new Linux installation, pass --with-supervisor to the installer script. To enable the OpAMP Supervisor after upgrading, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=true in /etc/otel/collector/splunk-otel-collector.conf and restart the service.

For a new Windows installation, pass -with_supervisor $true to the installer script. To enable the OpAMP Supervisor after upgrading, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=true in the service's Environment value under HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector and restart the service.

To stop running the Collector under OpAMP Supervisor, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=false in:

  • Linux: /etc/otel/collector/splunk-otel-collector.conf
  • Windows: service's Environment value under HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector

Restart the service after changing the setting. The launcher will return to starting otelcol directly, without the supervisor. When switched back to direct mode, remote configuration delivered through the supervisor is no longer applied.

On Linux DEB and RPM installation or upgrade, the package also now recursively sets the ownership of /var/lib/otelcol to the service user and group. This ensures the Collector and OpAMP Supervisor can write files in existing and new subdirectories of the shared state directory.

From 0.157.0 to 0.158.0

OpenTelemetry has deprecated deployment.environment in favor of deployment.environment.name. The Collector's optional deployment environment configuration and the --deployment-environment (Linux) and -deployment_env (Windows) installer options now use deployment.environment.name.

If deployment.environment is still required, use that name instead in the applicable configuration. For instrumented applications, set OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<value>. The Ansible, Chef, Puppet, and Salt integrations can pass the deprecated key through their generic resource-attribute option.

The installer deployment environment option will use the new attribute name. To keep the deprecated attribute, replace only that attribute in OTEL_RESOURCE_ATTRIBUTES, preserve all other attributes, and restart the instrumented applications or IIS.

From 0.150.0 to 0.151.0

The default Windows MSI artifact download URL has been updated: Splunk Observability Cloud domain transition guide

  • https://dl.signalfx.comhttps://dl.observability.splunkcloud.com

To keep using the legacy dl.signalfx.com download URL, pass -collector_msi_url explicitly to install.ps1 (replace <version> with the collector version):

Windows (install.ps1):

CODE
-collector_msi_url https://dl.signalfx.com/splunk-otel-collector/msi/release/splunk-otel-collector-<version>-amd64.msi

From 0.149.0 to 0.150.0

Default endpoint URLs have changed from *.signalfx.com to *.observability.splunkcloud.com. The legacy endpoints remaim functional but migrating to the new domain is highly recommended. See the domain transition guide for background.

Variable Old default New default
SPLUNK_API_URL https://api.<realm>.signalfx.com https://api.<realm>.observability.splunkcloud.com
SPLUNK_INGEST_URL https://ingest.<realm>.signalfx.com https://ingest.<realm>.observability.splunkcloud.com
SPLUNK_TRACE_URL https://ingest.<realm>.signalfx.com/v2/trace https://ingest.<realm>.observability.splunkcloud.com/v2/trace
SPLUNK_HEC_URL https://ingest.<realm>.signalfx.com/v1/log https://ingest.<realm>.observability.splunkcloud.com/v1/log
Note: If you have firewall rules or proxy allowlists scoped to *.signalfx.com, add *.observability.splunkcloud.com (or the realm-specific hostnames api.<realm>.observability.splunkcloud.com and ingest.<realm>.observability.splunkcloud.com) before switching.

Package manager upgrade (apt, yum, choco upgrade)

Your existing environment configuration is preserved. The collector continues to use the endpoints you previously configured.

When you are ready to migrate, replace with your realm (e.g. us0):

Linux: Edit /etc/otel/collector/splunk-otel-collector.conf:
CODE
SPLUNK_API_URL=https://api.<realm>.observability.splunkcloud.com
SPLUNK_INGEST_URL=https://ingest.<realm>.observability.splunkcloud.com
SPLUNK_HEC_URL=https://ingest.<realm>.observability.splunkcloud.com/v1/log

Then restart the service: sudo systemctl restart splunk-otel-collector.

Windows: Update the Environment value in the registry at HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector with the same URLs, then restart the service.

SPLUNK_TRACE_URL is derived automatically from SPLUNK_INGEST_URL at runtime unless you have explicitly set it. If you have, update it as well.

MSI upgrade (Windows)

Upgrading via msiexec applies the new *.observability.splunkcloud.com defaults. To keep the legacy endpoints, pass them as MSI properties:

CODE
msiexec /i <path\to\msi> SPLUNK_API_URL=https://api.<realm>.signalfx.com SPLUNK_INGEST_URL=https://ingest.<realm>.signalfx.com SPLUNK_HEC_URL=https://ingest.<realm>.signalfx.com/v1/log

New installation

New installations default to *.observability.splunkcloud.com. If your environment requires the legacy endpoints, pass them explicitly:

install.sh (add these flags):

CODE
--api-url https://api.<realm>.signalfx.com \
--ingest-url https://ingest.<realm>.signalfx.com

install.ps1 (add these flags):

CODE
-api_url https://api.<realm>.signalfx.com `
-ingest_url https://ingest.<realm>.signalfx.com

Chocolatey (add these params):

CODE
choco install splunk-otel-collector --params "'/SPLUNK_API_URL:https://api.<realm>.signalfx.com /SPLUNK_INGEST_URL:https://ingest.<realm>.signalfx.com /SPLUNK_HEC_URL:https://ingest.<realm>.signalfx.com/v1/log'"

msiexec (add these properties):

CODE
SPLUNK_API_URL=https://api.<realm>.signalfx.com SPLUNK_INGEST_URL=https://ingest.<realm>.signalfx.com SPLUNK_HEC_URL=https://ingest.<realm>.signalfx.com/v1/log

From 0.117.0 to 0.118.0

The deprecated syntax for config source expansion is no longer supported. Strings like $ENV or $include:/path/to/file.yaml will no longer be expanded. Instead, use the ${env:ENV} or ${include:/path/to/file.yaml} syntax. There are only two symbols allowed after $: { and $. The collector will log an error and fail to start if it encounters a bare config source. Please update your configuration files to use the correct syntax.

From 0.114.0 to 0.115.0

The sapm exporter still works as before but has been deprecated. Use the otlphttp exporter instead:

  1. Replace the sapm exporter with otlphttp exporter using the following configuration:

    YAML
    otlphttp:
      traces_endpoint: "${SPLUNK_INGEST_URL}/v2/trace/otlp"
      headers:
        "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}"
  2. Update traces pipeline to use otlphttp exporter instead of sapm:

    YAML
    service:
      pipelines:
        traces:
          exporters: [otlphttp, signalfx]

From 0.96.1 to 0.97.0

memory_ballast is no longer effective. You can now control garbage collection with a soft memory limit using the SPLUNK_MEMORY_TOTAL_MIB env var, which is set to 90% of the total memory by default. For more information, see Environment variables.

Follow these steps to ensure your Collector instances work correctly:

  • If you haven’t customized memory_ballast, remove it from the configuration.

  • If you have customized memory_ballast using SPLUNK_BALLAST_SIZE_MIB (or extensions::memory_ballast::size_mib config), remove the memory_ballast extension and use the GOMEMLIMIT environment variable to set a custom soft memory limit:

    • To increase frequency of garbage collection set GOMEMLIMIT to a higher value than the default 90% of total memory.

    • To decrease frequency of garbage collection set GOMEMLIMIT to a lower value than the default 90% of total memory.

    • For more information, see Go environment variables .

From 0.68.0 to 0.69.0

The gke and gce resource detectors in the resourcedetection processor have been replaced with the gcp resource detector. If you have gke and gce detectors configured in the resourcedetection processor, update your configuration accordingly.

For more information, see Resource detection processor.

From 0.41.0 to 0.42.0

The Splunk Distribution of the OpenTelemetry Collector used to evaluate user configuration twice and this required escaping of each $ symbol with $$ to prevent unwanted environment variable expansion. The issue was fixed in the 0.42.0 version. Any occurrences of $$ in your configuration need to be replaced with $.

From 0.35.0 to 0.36.0

Move the config parameter exporters -> otlp -> insecure to exporters -> otlp -> tls -> insecure.

The otlp exporter configuration must look like this:

YAML
exporters:
  otlp:
    endpoint: "${SPLUNK_GATEWAY_URL}:4317"
    tls:
      insecure: true

From 0.34.0 to 0.35.0

Move the ballast_size_mib parameter from the memory_limiter processor to the memory_ballast extension, and rename it to size_mib.

YAML
extensions:
  memory_ballast:
    size_mib: ${SPLUNK_BALLAST_SIZE_MIB}

Update the access token for the Collector for Kubernetes

Note: Make sure you don’t update your Helm chart or Collector version in the process of updating your access token. See Step 3 for details.

To update the access token for your Collector for Kubernetes instance follow these steps:

  1. Confirm the Helm release name and chart version. To do so, run:

    BASH
    helm list -f <Release_Name>
  2. Optionally, you can check your current access token:

    BASH
    helm get values <Release_Name>
  3. Deploy your new access token with Helm upgrade. This command will only update your access token, but will mantain your current Helm chart and Collector versions.

    BASH
    helm upgrade --reuse-values --version <Current_Chart_Version> --set splunkObservability.accessToken=<New_Access_Token> <Release_Name> splunk-otel-collector-chart/splunk-otel-collector

    If you want to use the latest Helm version instead of your current one, remove '--version <Current_Chart_Version>' from the command.

  4. Verify the value of the updated access token:

    BASH
    helm get values <Release_Name>
  5. Restart the Collector’s DaemonSet and deployments:

    • If agent.enabled=true, restart the Collector’s agent DaemonSet:

      BASH
      kubectl rollout restart DaemonSet <Release_Name>-agent
    • If clusterReceiver.enabled=true, restart the Collector’s cluster receiver deployment:

      BASH
      kubectl rollout restart deployment <Release_Name>-k8s-cluster-receiver
    • If gateway.enabled=true, restart the Collector’s gateway deployment:

      BASH
      kubectl rollout restart deployment <Release_Name>
  6. Verify the status of your clusters’ pods:

    BASH
    kubectl get pod -n <Namespace> | grep <Release_Name>