Collect Log Analytics Data from Syslog Messages

Related pages:

You can configure the Analytics Agent to receive syslog messages using TCP transport and standard syslog format adhering to RFC 5424. The Analytics Agent can listen on a TCP port for syslog messages coming from a remote source or from the same host on which the Analytics Agent is present.

To set this up:

Supported Environment Details

  • Linux only
  • Network protocol is TCP only
  • A template for Apache Web Server syslog format is shipped with the Analytics Agent. You can bring any log data in through syslog over TCP as long as you provide the correct configuration to parse the log message.
  • One job file per Analytics Agent can be enabled to receive syslog messages over TCP.
Note: Both the syslog utility and analytics-agent should have root access if the port where the analytics-agent is listening for syslog messages is lower than 1024.

Send syslog Data Directly to the Analytics Agent

You can use the Linux Logger utility to direct your Apache access and error log messages to the syslog daemon directly. Using this method, the logs are not written to the usual access log and error log apache files or to the /var/log/message file. Instead, the log lines are sent directly to a syslog daemon which then forwards the data to the analytics-agent.

Configure:

  1. Configure Apache to delegate logs to /usr/bin/logger .
    1. Locate and open httpd.conf , the Apache configuration file. This is typically located at /etc/httpd/conf/ .
    2. Add a new CustomLog directive to send access logs and error logs to the syslog and comment out the line that sends to the access_logs.
      #comment the following line to avoid logging to access_logs
#CustomLog logs/access_log combined
#Add a new CustomLog directive to send access logs and error logs to the syslog
CustomLog "|/usr/bin/logger -t httpd -p local6.info" combined
      This directive uses the logger utility to send messages with facility local6, tag 'httpd' and log format combined. The facility code specifies the type of program that is logging the message. Messages with different facilities may be handled differently.The tag 'httpd' in the httpd.conf directly relates to the program name in the rsyslog.conf (see next step) for filtering out which messages need to be sent. For example, there can be other programs writing to that particular port, but we only want to send the logs from the httpd program to analytics-agent.
  2. Configure the rsyslog client.
    1. Edit rsyslog.conf , typically located at /etc/ .
    2. Add the following lines above "RULES" or "var/log/messages" filter so Apache doesn't log to the /var/log/message file.
      # log to analytics-agent
if $syslogfacility-text == 'local6' and $programname == 'httpd' then @@<analytics_agent_home>:514
# Prevent logging httpd to /var/log/messages
if $syslogfacility-text == 'local6' and $programname == 'httpd' then ~
      Make sure you replace 514 with the port where the analytics-agent is listening for syslog messages. This must match the port specified in the job file.
  3. Restart apache and rsyslog, and look at /var/log/messages for any rsyslog errors.

Read syslog Messages From the Log File and Send to Analytics Agent

In this case, the Apache server writes logs to the usual access log and error log files, and the rsyslog daemon is configured to read from these log files and forward the log data to analytics agent. This method preserves the original log files. In this case, the access and the error log messages are logged in the /var/log/message file.

In this example, the rsyslog client is configured to read from a specific file and forward the message with facility local6 and severity level info over the specified port (default port is 514).

  1. Locate and edit rsyslog.conf, typically located at /etc/ .

  2. In the begin forwarding section of the rsyslog.conf file, add the following lines:

    # add these lines in the begin forwarding section
$ModLoad imfile
$InputFileName /etc/httpd/logs/access_log << your file
$InputFileTag apache-access
$InputFileStateFile stat-apache-access
$InputFileSeverity info
$InputFileFacility local6
$InputRunFileMonitor
local6.info @@localhost:514

    $InputFileName: path to the log file you want to tail.

    local6.info: Use the Analytics Agent IP address if your analytics agent is not local to the controller. If needed, replace 514 with the port where the analytics-agent is listening for syslog messages. This must match the port specified in the job file or the source rule.

  3. Restart rsyslog and review /var/log/messages for any rsyslog errors.

Configure Log Analytics TCP Source Rule

Using the Centralized Log Management UI, you can configure a source rule to extract log analytics fields from syslog messages over TCP.

  1. Access the Centralized Log Management UI from your Controller by clicking Analytics > Configuration > Log Analytics.
  2. On the Source Rules tab, click + Add.
  3. In the Add Source Rule panel, select the Create from source template and select From Network Connection as the collection type.Several log format templates are shipped with the Analytics Agent. You can create a new source rule for any log format over syslog TCP as long as you configure it correctly.
  4. Click Next to see the Add Source Configuration wizard.
  5. Specify the collection details, such as the name of the source rule, source type and enter the TCP Port where the Analytics Agent is listening.
  6. When you specify From Network Connection as the collection type, the grok pattern for the syslog header (which is appended to the log messages) is automatically added at the beginning of the grok Message Pattern: 
    %{SYSLOG5424PRI}%{SYSLOGBASE2}
  7. Confirm that the value for Multiline Format is None.
  8. Configure field extraction and field management as for any other source rule. See Configure Log Analytics Using Source Rules.

Configure Log Analytics TCP Job File

When selecting and configuring the port where the analytics agent will listen for the syslog data, make sure it does not conflict with anything else active in the network. If no port number is provided, port 514 is used. Both the syslog utility and analytics-agent should have root access to send logs to port 514 (binding to ports less than 1024 requires root access).

To allow the analytics-agent to listen at a port, specify the log file source property and associated parameters for type=syslog. For example, add the following to the appropriate job file:

source:
    type: syslog
    port: 514
    protocol: tcp
    numThreads: 1

A job file for apache commons is included in the analytics distribution at /<analytics-agent-home>/conf/job/sample-apache-httpserver-access-syslog.job.

Your job file should look similar to the following:

version: 2
enabled: true
source:
    type: syslog
    port: 514
    protocol: tcp
    numThreads: 5
fields:
    sourceType: apache-httpserver-access-syslog
    nodeName: Node1
    tierName: Tier1
    appName: App1
grok:
   patterns:
      - "%{SYSLOG5424PRI}%{SYSLOGBASE2} %{COMBINEDAPACHELOG}"
eventTimestamp:
   pattern: "dd/MMM/yyyy:HH:mm:ss Z"