Advanced configuration for Linux

Configure optional settings for the Splunk Distribution of OpenTelemetry Collector for Linux.

The Collector comes with a default configuration, as explained in Collector for Linux default configuration. Read on to modify advanced settings.

Send Linux metrics to the Splunk platform

Note: This guidance covers metrics collection with the Linux installer script.

Use the Linux installer script to collect host metrics with the Collector and send them to Splunk Enterprise or Splunk Cloud Platform.

Prerequisites

  • A running Splunk Enterprise or Splunk Cloud Platform deployment.

  • A Splunk HEC token with write access to the target metrics index. For more information, see Use HTTP Event Collector.

  • A metrics index in Splunk Enterprise or Splunk Cloud Platform. For more information, see Get started with metrics.

Install the Collector with metrics collection enabled

Pass the Splunk HEC token, endpoint URL, and target metrics index to the installer.

Note: --splunk-platform-url and --splunk-platform-token are required to activate Splunk Platform collection; --splunk-platform-metrics-index activates Splunk platform metrics collection and sets the destination index for metrics. All three options are required to activate metrics collection.
BASH 
curl -sSL https://dl.observability.splunkcloud.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh;
sudo sh /tmp/splunk-otel-collector.sh --splunk-platform-token "<your-hec-token>" \
  --splunk-platform-url "https://<your-splunk-host>:8088/services/collector" \
  --splunk-platform-metrics-index "<your-index>"

To also send metrics and traces to Splunk Observability Cloud, include your Splunk Observability Cloud access token and realm:

BASH 
curl -sSL https://dl.observability.splunkcloud.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh;
sudo sh /tmp/splunk-otel-collector.sh --realm <realm> \
  --splunk-platform-token "<your-hec-token>" \
  --splunk-platform-url "https://<your-splunk-host>:8088/services/collector" \
  --splunk-platform-metrics-index "<your-index>" -- <your-o11y-access-token>

Installer options

Use these installer options to send Linux metrics to the Splunk platform:

Option

Description

--splunk-platform-token <token>

Required. The Splunk HEC token that authenticates requests to the Splunk platform.

--splunk-platform-url <url>

Required. The Splunk HEC endpoint URL, such as https://splunk.example.com:8088/services/collector.

--splunk-platform-metrics-index <index>

Required. The Splunk metrics index to send metrics to.

Collected metrics

By default, the Collector uses the host_metrics receiver to collect system metrics from the Linux host every 10 seconds. The default scrapers collect CPU, disk, filesystem, memory, network, load, paging, and process-count metrics.

Setting or scraper Description
cpu CPU utilization and time metrics. See the CPU scraper documentation.
disk Disk I/O and operation metrics. See the disk scraper documentation.
filesystem Filesystem usage and capacity metrics. See the filesystem scraper documentation.
memory System memory usage metrics. See the memory scraper documentation.
network Network interface traffic and error metrics. See the network scraper documentation.
load System load average metrics. See the load scraper documentation.
paging Paging, swap-space utilization, and I/O metrics. See the paging scraper documentation.
processes Aggregated system process count metrics. See the processes scraper documentation.
process System process metrics. This scraper is turned off by default. See the process scraper documentation.

Activate or deactivate scrapers

  1. Edit /etc/otel/collector/splunk_metrics_config_linux.yaml and uncomment or comment scrapers in the host_metrics receiver:

    YAML 
    host_metrics:
  collection_interval: 10s
  scrapers:
    cpu:
    disk:
    filesystem:
    memory:
    network:
    load:
    paging:
    processes:
    #process:

    You can add other scrapers that the host_metrics receiver supports, such as nfs or system. For more information, see the host metrics receiver documentation.

    YAML 
    host_metrics:
  collection_interval: 10s
  scrapers:
    cpu:
    disk:
    filesystem:
    memory:
    network:
    load:
    paging:
    processes:
    nfs:
    system:

  2. Restart the Collector service:

    BASH 
    sudo systemctl restart splunk-otel-collector

Activate or deactivate individual metrics

You can activate or deactivate individual metrics within a scraper group. For example, to enable system.memory.linux.hugepages.reserved and system.memory.linux.hugepages.page_size, which are deactivated by default for the memory scraper, add the following configuration:

YAML 
host_metrics:
  collection_interval: 10s
  scrapers:
    memory:
      system.memory.linux.hugepages.reserved:
        enabled: true
      system.memory.linux.hugepages.page_size:
        enabled: true

To deactivate an individual metric that is activated by default, set enabled to false. For example, to deactivate system.cpu.load_average.15m:

YAML 
host_metrics:
  collection_interval: 10s
  scrapers:
    load:
      system.cpu.load_average.15m:
        enabled: false

Verify metrics ingestion

Run a metrics search in Splunk Cloud Platform or Splunk Enterprise to confirm that metrics are arriving in the target index:

CODE 
| mpreview index="<your-index>" | search source=otel

Change the default configuration file

For extended configuration options, see full_config_linux.yaml. This configuration requires OpenTelemetry Collector Contrib or a similar distribution.

After you modify the configuration, restart the Collector service. For example:

BASH 
sudo systemctl restart splunk-otel-collector

You can view the splunk-otel-collector service logs and errors in the systemd journal using the following command:

BASH 
sudo journalctl -u splunk-otel-collector
Note: See Tutorial: Configure the Splunk Distribution of OpenTelemetry Collector on a Linux host to learn how to configure the Collector.