Collect logs with the Collector for Linux

Collect Linux host logs with the Splunk Distribution of OpenTelemetry Collector and send them to the Splunk platform.

Note: The Splunk Universal Forwarder remains a supported method for sending logs to the Splunk platform. See Use the Splunk Universal Forwarder with the Collector.
Note: This guidance covers log collection with the Linux installer script.

Prerequisites

  • A running Splunk Enterprise or Splunk Cloud Platform deployment.

  • A Splunk HTTP Event Collector (HEC) token with write access to the target index. For more information, see Use HTTP Event Collector.

  • A Linux host where you can install the Splunk Distribution of OpenTelemetry Collector.

Install the Collector with log collection enabled

Pass the Splunk HEC token, endpoint URL, and target log index to the installer. Include --splunk-platform-logs-index so the Collector sends log events to a known index.

BASH 
curl -sSL https://dl.observability.splunkcloud.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh;
sudo sh /tmp/splunk-otel-collector.sh --splunk-platform-token "<your-hec-token>" \
  --splunk-platform-url "https://<your-splunk-host>:8088/services/collector" \
  --splunk-platform-logs-index "<your-index>"

To also send metrics and traces to Splunk Observability Cloud, include your Splunk Observability Cloud access token and realm:

BASH 
curl -sSL https://dl.observability.splunkcloud.com/splunk-otel-collector.sh > /tmp/splunk-otel-collector.sh;
sudo sh /tmp/splunk-otel-collector.sh --realm <realm> \
  --splunk-platform-token "<your-hec-token>" \
  --splunk-platform-url "https://<your-splunk-host>:8088/services/collector" \
  --splunk-platform-logs-index "<your-index>" -- <your-o11y-access-token>

Installer options

Use these installer options to send Linux logs to the Splunk platform:

Option

Description

--splunk-platform-token <token>

Required. The Splunk HEC token that authenticates requests to the Splunk platform.

--splunk-platform-url <url>

Required. The Splunk HEC endpoint URL, such as https://splunk.example.com:8088/services/collector.

--splunk-platform-logs-index <index>

Recommended. The Splunk index to send logs to. If you omit this option, the Splunk platform routes logs to the default index configured on the HEC token.

Collected logs

By default, the Collector tails files under /var/log that match common log patterns, such as *.log, *log, *auth*, *secure*, and *messages*. This behavior is similar to the Splunk Add-on for Unix and Linux.

For the full list of included paths and receivers, see splunk_logs_config_linux.yaml.

The configuration includes additional receivers that are deactivated by default:

Receiver

Description

file_log/etc

Configuration files under /etc. The receiver reads the files once each time the Collector starts.

file_log/bash_history

Bash history files.

journald

Systemd journal data. This receiver requires systemd journal group membership.

file_log/nginx-access, file_log/mysql-error, and other application-specific receivers

Application-specific log files. Experimental, subject to change.

Activate or deactivate receivers

  1. Edit /etc/otel/collector/splunk_logs_config_linux.yaml and uncomment receivers in service.pipelines.logs/hec.receivers:

    YAML 
    service:
  pipelines:
    logs/hec:
      receivers:
        - file_log/varlog
        #- file_log/etc
        #- file_log/bash_history
        #- journald
        #- file_log/nginx-access
        #- file_log/mysql-error

  2. Restart the Collector service:

    BASH 
    sudo systemctl restart splunk-otel-collector

Enable journald

The journald receiver requires the splunk-otel-collector user to be in the systemd-journal group. Without this group membership, the receiver starts but doesn't produce data.

BASH 
sudo usermod -aG systemd-journal splunk-otel-collector
sudo systemctl restart splunk-otel-collector

Add log files

Each file_log receiver has an include list of glob patterns. To collect logs from a custom application, add the application path to the file_log/varlog receiver:

YAML 
file_log/varlog:
  include:
    - /var/log/*.log
    - /var/log/*log
    - /opt/myapp/logs/*.log

To exclude files within a matched pattern, add the files to the exclude list:

YAML 
file_log/varlog:
  include:
    - /var/log/*.log
  exclude:
    - /var/log/noisy-app.log

For the full glob syntax and available options, see the filelog receiver documentation.

Verify log ingestion

Run a search in Splunk Cloud Platform or Splunk Enterprise for events that use the default receiver source type:

CODE 
index="<your-index>" sourcetype="linux:varlog"

To view all Linux source types currently being ingested, run the following search:

CODE 
index="<your-index>" sourcetype="linux:*" | stats count by sourcetype

Sizing

Log collection with the filelog receiver scales differently than metrics and traces. CPU usage depends mostly on the number of monitored files, not only total throughput.

Use the following CPU recommendations as a starting point for common file counts and per-file rates:

Files Lines per second per file Total throughput Recommended CPUs
<= 100 100 10,000 lines per second 0.5
<= 500 100 50,000 lines per second 1
<= 1,000 100 100,000 lines per second 2
<= 50 1,000 45,000 lines per second 0.5
<= 100 1,000 90,000 lines per second 1
<= 150 1,000 135,000 lines per second 2

These values were tested with plain-text log lines and no parsing or field extraction in the Collector pipeline. In these tests, log line size was not the main driver of resource usage. Most Collector resources were spent monitoring files and processing records.

Note: Memory is not the primary sizing constraint for filelog. With block_on_overflow: true, the receiver pauses reading when the queue is full instead of increasing memory usage indefinitely. However, the default memory limit of 512 MB can be too small under higher load. Increase SPLUNK_MEMORY_LIMIT_MIB when the memory_limiter processor starts refusing new data.

At equal events per second, more files require more CPU. The following examples show observed CPU usage from the source sizing tests:

Scenario Total throughput Average CPU observed
10 files at 1,000 lines per second About 10,000 lines per second About 9%
100 files at 100 lines per second About 10,000 lines per second About 14%
50 files at 1,000 lines per second About 45,000 lines per second About 39%
500 files at 100 lines per second About 50,000 lines per second About 84%

  • CPU scales approximately linearly with throughput for a similar file shape and pipeline configuration.

  • Using multiple filelog receivers can reduce CPU usage at high file counts.

  • For vertical scaling, Splunk HEC and the indexing tier are often the limiting factors. Size HEC and the indexing tier for the expected load so retries and backpressure don't increase Collector resource usage.

  • JSON parsing, regex operators, and similar processing stages increase CPU usage above these baseline values.

Validate these starting values against your log shape, file count, and downstream capacity.

Troubleshoot log collection

See Troubleshoot log collection.