Troubleshoot OpAmp Supervisor

Use the following sections to troubleshoot the issues that you might encounter with OpAmp Supervisor.

Troubleshoot remote configuration that is not applied

Check the server, OpAMP Supervisor, and Collector state to identify where a remote configuration deployment stopped.

If a remote config isn't being applied, use the following steps to identify where the deployment has stopped:
  1. In the group details page, confirm that the group includes the Collector.

    If the group doesn't include the Collector, the server hasn't targeted that Collector with the remote configuration.

  2. Check whether the deployment status changed for the Collector.

    The final Applied status can take up to 2 minutes to appear.

    If the deployment status didn't change, the server hasn't sent the remote configuration to the Collector.

  3. Confirm that OpAMP Supervisor received the remote configuration.

    Look for one or more of the following indicators:

    • The Received remote config from server message in the debug logs. To turn on debug logging, see Configure OpAMP Supervisor.
    • An updated last_recv_remote_config.dat or persistent_state.yaml file in the Supervisor state directory:
      • Linux: /var/lib/otelcol/supervisor
      • Windows: C:\ProgramData\Splunk\OpenTelemetry Collector\supervisor

    If the log message and updated state files aren't present, the Collector hasn't received the remote configuration message.

  4. Check whether the Collector failed to apply the configuration.

    A Failed deployment status indicates an apply failure. Review the logs for the underlying validation or startup error.

  5. If the deployment status displays as Applied but the Collector configuration behaves unexpectedly, verify your command-line flags. Flags such as --discovery and --set override remote configuration settings by taking precedence.
The checks identify whether targeting, delivery, receipt, validation, or startup prevented the remote configuration from being applied.

Configure OpAMP Supervisor

Edit a supported OpAMP Supervisor setting and restart the Collector service.

OpAMP Supervisor configuration file is in the following platform-specific location:

Platform File path
Linux /etc/otel/collector/supervisor/supervisor_config.yaml
Windows C:\ProgramData\Splunk\OpenTelemetry Collector\supervisor\supervisor_config.yaml
Note: OpAMP Supervisor manages the agent.executable, agent.config_files, and agent.args settings automatically. It ignores changes to these settings in supervisor_config.yaml.
Warning: Don't edit supervisor_runtime_config.yaml. OpAMP Supervisor generates this file automatically and might overwrite it.

To change a supported setting of OpAMP Supervisor:

  1. Open supervisor_config.yaml in a text editor.
  2. Change a supported Supervisor setting.

    For example, use the following configuration to change the log level:

    CODE 
    telemetry:
  logs:
    level: debug

    For all upstream settings, see OpAMP Supervisor configuration options.

  3. Restart the Collector service.

Collector stops or repeatedly restarts after a remote update

Correct and redeploy a remote configuration that stops the Collector or causes repeated restarts.

The service can remain running while OpAMP Supervisor restarts the Collector process. One reload is expected when OpAMP Supervisor applies an update. Repeated restarts or a Collector that remains stopped indicate a problem with the new configuration.

  • Before OpAMP Supervisor applies an update, it runs otelcol validate against the combined local and remote configuration. If validation fails, OpAMP Supervisor doesn't apply the update. Validation can't detect problems that occur only after the Collector starts.
  • OpAMP Supervisor saves the last remote configuration that it receives. It might try to apply the same problematic configuration after the service restarts or after Supervisor mode is turned on again.

To resolve the issue:

  1. Correct the remote configuration.
  2. Deploy the corrected remote configuration again.
  3. Optional: If you are unable to deploy a new group with the corrected remote configuration after deleting an existing group, remove last_recv_remote_config.dat from the Supervisor state directory and restart the service. This process clears the problematic configuration and allows the Collector to restart successfully.
    • Linux: /var/lib/otelcol/supervisor
    • Windows: C:\ProgramData\Splunk\OpenTelemetry Collector\supervisor
The Supervisor can apply the corrected remote configuration without repeatedly restarting the Collector.

Collector service stops after enabling OpAMP Supervisor

Collector configuration and provider URI requirements can prevent the service from starting under the OpAMP Supervisor.

If the Collector service stops after you enable OpAMP Supervisor, check the service logs for launcher or startup errors.

Common causes include the following conditions:

  • No Collector configuration was provided through the SPLUNK_CONFIG environment variable or a --config service argument.
  • The top-level configuration uses an unsupported provider URI. Supervisor mode supports local file paths and file: URIs. It doesn't support env:, http:, or https: sources.