Troubleshoot log collection

Describes known issues when collecting logs with the Splunk Distribution of the OpenTelemetry Collector.

This document describes common issues related to log collection with the Splunk Distribution of the OpenTelemetry Collector.

To troubleshoot the health and performance of the Collector see the OpenTelemetry Project troublehooting docs . It includes information about troubleshooting tools and debugging.

For more information on log collection see:

Platform

Docs

Kubernetes

Collect logs and events with the Collector for Kubernetes

Linux

Collect logs with the Collector for Linux

Windows

Collect logs with the Collector for Windows

A source isn’t generating logs

Run the following command(s) to confirm that your source is generating logs:

Linux
BASH 
tail -f /var/log/myTestLog.log
journalctl -u my-service.service -f
Windows
SHELL 
Get-Content myTestLog.log

The Collector isn’t configured properly

Do the following to check the Collector configuration:

  • Go to http://localhost:55679/debug/tracez to check zPages for samples. You might need to configure the endpoint.

  • Activate the debug exporter.

  • Run journalctl -u splunk-otel-collector.service -f to collect the logs for you to review.

  • Review Troubleshoot the Splunk OpenTelemetry Collector if you can’t find what you need in the logs.

  • Generate logs manually by running these commands and see if the Collector sends them:

    BASH 
    echo "2021-03-17 02:14:44 +0000 [debug]: test" >>/var/log/syslog.log
echo "2021-03-17 02:14:44 +0000 [debug]: test" | systemd-cat

Unwanted profiling logs appear in Splunk Observability Cloud

By default, the Collector sends AlwaysOn Profiling data using the Splunk HEC exporter. To deactivate this see Turn off logs or profiling data.

Exclude log data in the Collector

Depending on its configuration, the Collector might collect and send logs to Splunk Observability Cloud through a logs pipeline that uses the Splunk HEC exporter. To deactivate this see Turn off logs or profiling data.

Send logs from the Collector to Splunk Cloud Platform or Splunk Enterprise

See Send logs to Splunk Cloud Platform or Splunk Enterprise.