Upgrade the Collector for Kubernetes and other updates
Upgrade the Splunk Distribution of OpenTelemetry Collector for Kubernetes.
Upgrade the Collector for Kubernetes
The installer script uses one of the supported package managers to install the Collector. When you update the Collector using the official packages, configuration files are never overridden. If you need to update the configuration after an update, edit them manually before backward compatibility is dropped.
To upgrade the Collector for Kubernetes run the following commands:
-
Use the flag
--reuse-valuesto keep the config values you’d already set while installing or using the Collector:
helm upgrade splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector
--reuse-values
-
Use
--values config.yamlto override your previous configuration while upgrading:
helm upgrade splunk-otel-collector --values config.yaml splunk-otel-collector-chart/splunk-otel-collector --reuse-values
Read more in the official Helm upgrade options documentation.
Upgrade guidelines
Apply the following changes to the Collector configuration files for specific version upgrades. For more details refer to Helm chart upgrade guidelines in GitHub.
From 0.113.0 to 0.116.0
Custom resource definition (CRD) configuration has been modified.
-
Before v0.110.0 CRDs were deployed via a
crds/directory (upstream default). -
From v0.110.0 to v1.113.0 CRDs were deployed using Helm templates (upstream default), which had reported issues.
-
From v0.116.0 and higher, you must explicitly configure your preferred CRD deployment method or deploy the CRDs manually to avoid potential issues. You can deploy CRDs via a
crds/directory again by enabling a newly added value.
New users
If you’re a new user deploy CRDs via the crds/ directory. For a fresh installation use the following Helm values:
operatorcrds:
install: true
operator:
enabled: true
To install the chart run:
helm install <release-name> splunk-otel-collector-chart/splunk-otel-collector --set operatorcrds.install=true,operator. enabled=true <extra_args>
Current users
You might need to migrate if using operator.enabled=true.
If you’re using versions 0.110.0 to 1.113.0, CRDs are likely deployed via Helm templates. To migrate to the recommended crds/ directory deployment:
-
Delete the existing chart running
BASHhelm delete <release-name> -
Verify if the following CRDs are present and delete them if necessary:
BASHkubectl get crds | grep opentelemetry kubectl delete crd opentelemetrycollectors.opentelemetry.io kubectl delete crd opampbridges.opentelemetry.io kubectl delete crd instrumentations.opentelemetry.io -
Reinstall the chart with the updated configuration:
BASHhelm install <release-name> splunk-otel-collector --set operatorcrds.install=true,operator.enabled=true <extra_args>
Current users maintaining legacy templates
If you’re using chart versions 0.110.0 to 1.113.0 and prefer to continue deploying CRDs via Helm templates (not recommended), use the following values:
operator:
enabled: true
operator:
crds:
create: true
From 0.105.5 to 0.108.0
.Values.operator.instrumentation.spec.* no migration is required.
The Helm chart configuration for operator auto-instrumentation has been simplified, and the values previously under .Values.operator.instrumentation.spec.* have been moved to .Values.instrumentation.*.
The updated path looks like this:
instrumentation:
endpoint: XXX
...
The deprecated path was:
operator:
instrumentation:
spec:
endpoint: XXX
...
The installer script uses one of the supported package managers to install the Collector.
When you update the Collector using the official packages, configuration files are never overridden. If you need to update the configuration after an update, edit them manually before backward compatibility is dropped.
General guidelines
Apply the following changes to the Collector configuration files for specific version upgrades.
From 0.158.0 to 0.159.0
Linux DEB and RPM packages and Windows installations through MSI or Chocolatey now use otelcollauncher as the service entrypoint instead of otelcol. By default, the launcher starts otelcol directly and passes through the existing service arguments, so upgrades preserve the previous Collector service behavior. The launcher allows the service to start either otelcol directly or the OpAMP Supervisor, enabling additional Fleet Management capabilities. See OpenTelemetry Fleet Management for details.
For a new Linux installation, pass --with-supervisor to the installer script. To enable the OpAMP Supervisor after upgrading, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=true in /etc/otel/collector/splunk-otel-collector.conf and restart the service.
For a new Windows installation, pass -with_supervisor $true to the installer script. To enable the OpAMP Supervisor after upgrading, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=true in the service's Environment value under HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector and restart the service.
To stop running the Collector under OpAMP Supervisor, set SPLUNK_OPAMP_SUPERVISOR_ENABLED=false in:
- Linux:
/etc/otel/collector/splunk-otel-collector.conf - Windows: service's
Environmentvalue underHKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector
Restart the service after changing the setting. The launcher will return to starting otelcol directly, without the supervisor. When switched back to direct mode, remote configuration delivered through the supervisor is no longer applied.
On Linux DEB and RPM installation or upgrade, the package also now recursively sets the ownership of /var/lib/otelcol to the service user and group. This ensures the Collector and OpAMP Supervisor can write files in existing and new subdirectories of the shared state directory.
From 0.157.0 to 0.158.0
OpenTelemetry has deprecated deployment.environment in favor of deployment.environment.name. The Collector's optional deployment environment configuration and the --deployment-environment (Linux) and -deployment_env (Windows) installer options now use deployment.environment.name.
If deployment.environment is still required, use that name instead in the applicable configuration. For instrumented applications, set OTEL_RESOURCE_ATTRIBUTES=deployment.environment=<value>. The Ansible, Chef, Puppet, and Salt integrations can pass the deprecated key through their generic resource-attribute option.
The installer deployment environment option will use the new attribute name. To keep the deprecated attribute, replace only that attribute in OTEL_RESOURCE_ATTRIBUTES, preserve all other attributes, and restart the instrumented applications or IIS.
From 0.150.0 to 0.151.0
The default Windows MSI artifact download URL has been updated: Splunk Observability Cloud domain transition guide
https://dl.signalfx.com→https://dl.observability.splunkcloud.com
To keep using the legacy dl.signalfx.com download URL, pass -collector_msi_url explicitly to install.ps1 (replace <version> with the collector version):
Windows (install.ps1):
-collector_msi_url https://dl.signalfx.com/splunk-otel-collector/msi/release/splunk-otel-collector-<version>-amd64.msi
From 0.149.0 to 0.150.0
Default endpoint URLs have changed from *.signalfx.com to *.observability.splunkcloud.com. The legacy endpoints remaim functional but migrating to the new domain is highly recommended. See the domain transition guide for background.
| Variable | Old default | New default |
|---|---|---|
SPLUNK_API_URL |
https://api.<realm>.signalfx.com |
https://api.<realm>.observability.splunkcloud.com |
SPLUNK_INGEST_URL |
https://ingest.<realm>.signalfx.com |
https://ingest.<realm>.observability.splunkcloud.com |
SPLUNK_TRACE_URL |
https://ingest.<realm>.signalfx.com/v2/trace |
https://ingest.<realm>.observability.splunkcloud.com/v2/trace |
SPLUNK_HEC_URL |
https://ingest.<realm>.signalfx.com/v1/log |
https://ingest.<realm>.observability.splunkcloud.com/v1/log |
*.signalfx.com, add *.observability.splunkcloud.com (or the realm-specific hostnames api.<realm>.observability.splunkcloud.com and ingest.<realm>.observability.splunkcloud.com) before switching.
Package manager upgrade (apt, yum, choco upgrade)
Your existing environment configuration is preserved. The collector continues to use the endpoints you previously configured.
When you are ready to migrate, replace with your realm (e.g. us0):
Linux: Edit /etc/otel/collector/splunk-otel-collector.conf:
SPLUNK_API_URL=https://api.<realm>.observability.splunkcloud.com
SPLUNK_INGEST_URL=https://ingest.<realm>.observability.splunkcloud.com
SPLUNK_HEC_URL=https://ingest.<realm>.observability.splunkcloud.com/v1/log
Then restart the service: sudo systemctl restart splunk-otel-collector.
Windows: Update the Environment value in the registry at HKLM:\SYSTEM\CurrentControlSet\Services\splunk-otel-collector with the same URLs, then restart the service.
SPLUNK_TRACE_URL is derived automatically from SPLUNK_INGEST_URL at runtime unless you have explicitly set it. If you have, update it as well.
MSI upgrade (Windows)
Upgrading via msiexec applies the new *.observability.splunkcloud.com defaults. To keep the legacy endpoints, pass them as MSI properties:
msiexec /i <path\to\msi> SPLUNK_API_URL=https://api.<realm>.signalfx.com SPLUNK_INGEST_URL=https://ingest.<realm>.signalfx.com SPLUNK_HEC_URL=https://ingest.<realm>.signalfx.com/v1/log
New installation
New installations default to *.observability.splunkcloud.com. If your environment requires the legacy endpoints, pass them explicitly:
install.sh (add these flags):
--api-url https://api.<realm>.signalfx.com \
--ingest-url https://ingest.<realm>.signalfx.com
install.ps1 (add these flags):
-api_url https://api.<realm>.signalfx.com `
-ingest_url https://ingest.<realm>.signalfx.com
Chocolatey (add these params):
choco install splunk-otel-collector --params "'/SPLUNK_API_URL:https://api.<realm>.signalfx.com /SPLUNK_INGEST_URL:https://ingest.<realm>.signalfx.com /SPLUNK_HEC_URL:https://ingest.<realm>.signalfx.com/v1/log'"
msiexec (add these properties):
SPLUNK_API_URL=https://api.<realm>.signalfx.com SPLUNK_INGEST_URL=https://ingest.<realm>.signalfx.com SPLUNK_HEC_URL=https://ingest.<realm>.signalfx.com/v1/log
From 0.117.0 to 0.118.0
The deprecated syntax for config source expansion is no longer supported. Strings like $ENV or $include:/path/to/file.yaml will no longer be expanded. Instead, use the ${env:ENV} or ${include:/path/to/file.yaml} syntax. There are only two symbols allowed after $: { and $. The collector will log an error and fail to start if it encounters a bare config source. Please update your configuration files to use the correct syntax.
From 0.114.0 to 0.115.0
The sapm exporter still works as before but has been deprecated. Use the otlphttp exporter instead:
-
Replace the
sapmexporter withotlphttpexporter using the following configuration:YAMLotlphttp: traces_endpoint: "${SPLUNK_INGEST_URL}/v2/trace/otlp" headers: "X-SF-Token": "${SPLUNK_ACCESS_TOKEN}" -
Update
tracespipeline to useotlphttpexporter instead ofsapm:YAMLservice: pipelines: traces: exporters: [otlphttp, signalfx]
From 0.96.1 to 0.97.0
memory_ballast is no longer effective. You can now control garbage collection with a soft memory limit using the SPLUNK_MEMORY_TOTAL_MIB env var, which is set to 90% of the total memory by default. For more information, see Environment variables.
Follow these steps to ensure your Collector instances work correctly:
-
If you haven’t customized
memory_ballast, remove it from the configuration. -
If you have customized
memory_ballastusingSPLUNK_BALLAST_SIZE_MIB(orextensions::memory_ballast::size_mib config), remove thememory_ballastextension and use theGOMEMLIMITenvironment variable to set a custom soft memory limit:-
To increase frequency of garbage collection set
GOMEMLIMITto a higher value than the default 90% of total memory. -
To decrease frequency of garbage collection set
GOMEMLIMITto a lower value than the default 90% of total memory. -
For more information, see Go environment variables .
-
From 0.68.0 to 0.69.0
The gke and gce resource detectors in the resourcedetection processor have been replaced with the gcp resource detector. If you have gke and gce detectors configured in the resourcedetection processor, update your configuration accordingly.
For more information, see Resource detection processor.
From 0.41.0 to 0.42.0
The Splunk Distribution of the OpenTelemetry Collector used to evaluate user configuration twice and this required escaping of each $ symbol with $$ to prevent unwanted environment variable expansion. The issue was fixed in the 0.42.0 version. Any occurrences of $$ in your configuration need to be replaced with $.
From 0.35.0 to 0.36.0
Move the config parameter exporters -> otlp -> insecure to exporters -> otlp -> tls -> insecure.
The otlp exporter configuration must look like this:
exporters:
otlp:
endpoint: "${SPLUNK_GATEWAY_URL}:4317"
tls:
insecure: true
From 0.34.0 to 0.35.0
Move the ballast_size_mib parameter from the memory_limiter processor to the memory_ballast extension, and rename it to size_mib.
extensions:
memory_ballast:
size_mib: ${SPLUNK_BALLAST_SIZE_MIB}
Update the access token for the Collector for Kubernetes
To update the access token for your Collector for Kubernetes instance follow these steps:
-
Confirm the Helm release name and chart version. To do so, run:
BASHhelm list -f <Release_Name> -
Optionally, you can check your current access token:
BASHhelm get values <Release_Name> -
Deploy your new access token with Helm upgrade. This command will only update your access token, but will mantain your current Helm chart and Collector versions.
BASHhelm upgrade --reuse-values --version <Current_Chart_Version> --set splunkObservability.accessToken=<New_Access_Token> <Release_Name> splunk-otel-collector-chart/splunk-otel-collectorIf you want to use the latest Helm version instead of your current one, remove
'--version <Current_Chart_Version>'from the command. -
Verify the value of the updated access token:
BASHhelm get values <Release_Name> -
Restart the Collector’s DaemonSet and deployments:
-
If
agent.enabled=true, restart the Collector’s agent DaemonSet:BASHkubectl rollout restart DaemonSet <Release_Name>-agent -
If
clusterReceiver.enabled=true, restart the Collector’s cluster receiver deployment:BASHkubectl rollout restart deployment <Release_Name>-k8s-cluster-receiver -
If
gateway.enabled=true, restart the Collector’s gateway deployment:BASHkubectl rollout restart deployment <Release_Name>
-
-
Verify the status of your clusters’ pods:
BASHkubectl get pod -n <Namespace> | grep <Release_Name>