Monitoring the status of OpenTelemetry Collectors in Splunk Enterprise

This read-only capability of OTel Collectors fleet overview allows you to monitor the health and status of your entire fleet of OpenTelemetry Collectors. It enables you to manage and troubleshoot your agents. This functionality is part of agent management and will be expanded in the future.

OTel Collectors is an open standard for collecting traces, logs, and metrics. For more information, see https://opentelemetry.io/docs/collector/.

OTel Collectors connect to the agent management using the OpenTelemetry Agent Management Protocol (OpAMP). This open standard enables centralized communication and status reporting. For more information about OpAMP, see https://opentelemetry.io/docs/specs/opamp/.

Turn on OpenTelemetry Collectors in Splunk Enterprise

You can view OTel Collectors after connecting them to Splunk Enterprise. To configure OTel Collectors, perform the following tasks:

  1. Configure Splunk Enterprise and enable OpenTelemetryTel Collectors
  2. Configure open-source OTel Collector

Before you begin

To use OTel Collectors, make sure you fulfill the following requirements:

  • Use Splunk Enterprise 10.0.0 or higher.
  • Use the admin role.
  • Use agent management on Linux on AMD64.
Note: If you want to use OTel Collectors fleet overview capability, Splunk Enterprise must run on Linux with the AMD64 architecture. OTel Collectors can run on any architecture supported by the OpenTelemetry project.

Configure Splunk Enterprise and turn on OpenTelemetry Collectors

To be able to use the OTel Collectors feature in agent management, you have to configure Splunk Enterprise. This topic presents an example of configuration.

The following configuration in this topic is only an example.

  1. Log in to Splunk Enterprise as an admin.
  2. Enable the feature using one of the following options:
    1. In Splunk Enterprise, follow these steps:
      1. Select the settings icon (Settings), and then under the Distributed Environment section, select Agent management.

        The Agent management page opens.

      2. On the left menu, select the OTel Collectors view.

      3. If you use this feature for the first time, you have to turn on OTel Collectors management. Select Enable OTel Collectors management.
      4. Select Restart Splunk. The restart might take some time.
    2. In the server.conf file, follow these steps:
      1. Use the following settings:
        CODE 
        [data_management]
otel_collector_management_enabled = true
      2. Restart Splunk Enterprise.
  3. After turning on the feature, refresh the page to access the empty list of OTel Collectors.
  4. Copy the IP address of your Agent Management Splunk server for later use.
  5. Set up access tokens:
    1. Log in to Splunk Enterprise as an admin.
    2. Select the settings icon (Settings) and under the Users and Authentication section, select Token, and then select New Token. For more information about token creation, see Create authentication tokens in the Securing Splunk Enterprise manual.
    3. Create a new token, such as with the following settings:
      • User = admin
      • Audience = general
      • Expiration = +30 d
      Note: The default value for Expiration is +30 d. It means that the token expires after 30 days.
      Important: The user for whom you create the token must have at least the data_management_agent role.
    4. Select Create.
    5. The token is displayed in the Token field. Copy the token and paste it into the config.yaml configuration file as described in the next section.
    6. Select Close.

Next step

Configure an OpenTelemetry Collector to connect to agent management

Configure an OpenTelemetry Collector to connect to agent management

Configure an OpenTelemetry Collector to connect to agent management

You can connect either the Splunk distribution of the OpenTelemetry Collector or the upstream open-source OpenTelemetry Collector Contrib to agent management. The Splunk distribution already includes the OpAMP extension. If you use the open-source Collector Contrib, the OpAMP extension is also included.

  1. On your machine prepare a plain text configuration file named config.yaml. Replace the placeholders for hostname or IP address and token with the data prepared in Configure Splunk Enterprise and turn on OpenTelemetry Collectors.

    Use the hostname or IP address and management port of agent management you want the collector to connect with.

    CODE 
    receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      memory:

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      exporters: [debug]
  extensions: [opamp]

extensions:
  opamp:
    server:
      http:
        endpoint: https://<hostname_or_IP>:<management_port>/services/tenant/agent-management/v2/opamp/otel
        tls:
          ca_file: /path/to/ca-cert.pem
        headers:
          Authorization: Bearer <your_authentication_token>
    Tip: If you use self-signed certificates, provide the path to your Certificate Authority (CA) file in the ca_file field. Avoid setting insecure_skip_verify: true in production environments, as it disables certificate verification and exposes the connection to potential man-in-the-middle attacks. Use insecure_skip_verify: true only in development or testing environments.
  2. Copy your file to your target machine using the following command:
    CODE 
    scp config.yaml <user>@<hostname_or_IP>:/home/<user>
  3. Download the OpenTelemetry Collector to your Linux machine.

    If you use the Splunk distribution of the OpenTelemetry Collector, see the Splunk OpenTelemetry Collector documentation for installation instructions. The OpAMP extension is already included.

    If you use the open-source Collector Contrib, download it from the OpenTelemetry Collector GitHub repository: https://github.com/open-telemetry/opentelemetry-collector-releases/releases.

    For example, you can use the following command to download otelcol-contrib_0.124.1_linux_amd64.tar.gz for AMD CPU Architecture:
    CODE 
    curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.124.1/otelcol-contrib_0.124.1_linux_amd64.tar.gz
  4. If you downloaded the open-source Collector Contrib archive, untar the file with the following command:
    CODE 
    tar -xvf otelcol-contrib_0.124.1_linux_amd64.tar.gz
  5. Run the OTel Collector using your configuration file:
    CODE 
    ./otelcol-contrib --config=config.yaml
  6. Now, you can see the OTel Collector registered in the agent management. Select the settings icon (Settings), and then select Agent management.
  7. On the Agent management page, on the menu on the left, select the OTel Collectors view.

Next step

After you configure and connect OTel Collectors to your Splunk Enterprise instance, you can view them on Splunk Enterprise. See View OpenTelemetry Collectors in Splunk Enterprise

View OpenTelemetry Collectors in Splunk Enterprise

After configuring and connecting OTel Collectors to your Splunk Enterprise instance, you can view them in a table. You can monitor their status and check the details.

  1. Log in to Splunk Enterprise.
  2. Select the settings icon (Settings), and then under the Distributed Environment section, select Agent management.

    The Agent management page opens.

  3. On the left menu, select the OTel Collectors view.
  4. On the Agent management - OTel Collectors page, you can see the list of OTel Collector. Information about OTel Collectors are grouped into the following table columns:
    Parameters Description
    Agent ID The ID identifying the agent (service.instance.id)
    Service Name Agent name (service.name)
    Namespace Agent namespace (service.namespace)
    Version Agent application version (service.version)
    Host Name Agent host name (host.name)
    IP Address Agent IP address
    CPU Architecture The CPU architecture the host system is running on (host.arch)
    System Description Human-readable OS version information (os.description)
    Status The status of an agent. The possible values are OK and Offline. The status is calculated on the basis of heartbeat
    Check-in Shows how long ago the last heartbeat of agent was sent
  5. To see the details, select the link in the Agent ID column. It displays a page with agent details, host detail, identifying attributes, and non-identifying attributes.
    • Agent details show Capabilities that are all the functions supported by a given agent.
    • Identifying attributes are attributes that identify the agent.
    • Non-identifying attributes are attributes that don't necessarily identify the agent but they help describe where it runs.
      The agent details and information about attributes

Delete OpenTelemetry Collector agents in Splunk Enterprise

To keep the list of agents tidy, you can remove agents that are offline. However, if you remove an agent with the OK status, it shows up again the next time it sends a heartbeat.

  1. Log in to Splunk Enterprise.
  2. Select the settings icon (Settings), and then under the Distributed Environment section, select Agent management.

    The Agent management page opens.

  3. On the left menu, select the OTel Collectors view.
  4. On the Agent management - OTel Collectors page, select the check box next to the OTel Collector that you want to delete and select Delete.
  5. Confirm the deletion.

    The OTel Collector is removed from the list.