Application-layer authentication between forwarders and indexers

Learn about the application-layer authentication exchange between forwarders and indexers designed to verify identity in addition to Transport Layer Security (TLS).

Application-layer authentication strengthens the security of Splunk-to-Splunk (S2S) communication by adding an authentication exchange between a forwarder and an indexer. The exchange operates at the application layer, above the transport security provided by Transport Layer Security (TLS), and provides additional assurance about the identities of the participants in each S2S connection.

Application-layer authentication operates in audit mode by default. No additional configuration is required on the forwarder or indexer to use this mode. In audit mode, Splunk software validates and reports authentication outcomes without interrupting data flow or affecting existing ingestion workflows. The authentication outcome for each connection is recorded in logs.

Benefits of application-layer authentication

Application-layer authentication provides the following benefits:

  • Additional authentication assurance - Provides assurance about the identities of the forwarder and indexer in addition to the transport-level security provided by TLS.

  • Connection-specific authentication - Provides an independent authentication outcome for each forwarder-to-indexer connection.

  • Auditability and monitoring - Records authentication outcomes and metrics that administrators can use to monitor authentication across S2S connections.

How application-layer authentication works

Each forwarder-to-indexer connection performs an independent authentication exchange. This includes new connections that a forwarder establishes when automatically load balancing data across receivers.

During connection establishment, the forwarder and indexer exchange random values and derive a session key from embedded keying material.

The authentication exchange is bound to the negotiated TLS session so that authentication is specific to that connection. Splunk software uses exported keying material (EKM) to bind the exchange to the TLS session. The RFC 5705 standard defines a TLS exporter mechanism that lets both sides derive the same EKM from the TLS connection for use by the application-layer authentication exchange.

Security assurance and monitoring

Splunk software logs S2S authentication outcomes and metrics on both indexers and forwarders so that administrators can monitor authentication exchanges without exposing sensitive keying material.

The authentication outcome for each forwarder-to-indexer connection indicates whether the application-layer authentication exchange completed successfully. Splunk software records these outcomes on both sides of the connection to internal logs.

On the indexer, authentication outcomes are logged by the TcpInputProc component. On the forwarder, authentication outcomes are logged by the AutoLoadBalancedConnectionStrategy component.

Log entries provide information about the authentication exchange, including authentication outcomes and, depending on the event, details such as the receiving host, authentication mode, and negotiated protocol version. Some information, including failure reasons, can appear as prose in the log message rather than as structured fields.

Note: Sensitive authentication data, including keying material, derived keys, and proof values, isn't written to logs. The logs contain only metadata about the authentication exchange.

Splunk software records S2S authentication metrics in metrics.log under group=s2s_auth. Administrators can use these metrics to track authentication handshake success rates and identify connections that haven't completed the authentication exchange.

Verify the authentication exchange

Verify application-layer authentication by checking S2S log messages and metrics.

To verify that the authentication exchange completes successfully, review the S2S authentication metrics in metrics.log.

In a healthy deployment where all applicable authentication exchanges succeed, s2s_auth.handshake_success is greater than zero and handshake_success_rate is 100%.

To investigate individual connections, review the application-layer authentication log messages. The following table describes the possible authentication outcomes and their corresponding log messages.

Outcome Message What it means
Successful authentication S2S auth validated src=<source> mode=<authMode> ...

The authentication exchange completed successfully.

This DEBUG-level message appears only when debug logging is turned on for the relevant component. For information on how to turn on debug logging, see Enable debug logging.
Forwarder doesn't support application-layer authentication Forwarder did not advertise S2S auth support

The forwarder didn't support application-layer authentication.
Indexer doesn't support application-layer authentication S2S auth: indexer did not advertise support

The indexer didn't support application-layer authentication.
Authentication exchange failed S2S auth failure, continuing src=<source> reason=<reason> mode=<mode> <handshake-fields...>

The authentication exchange failed. In audit mode, Splunk software logs the failure and continues forwarding data.
Indexer authentication proof verification failed in audit mode S2S auth: handshake failed, continuing idxHost=<ip> reason=indexer proof verification failed mode=<mode> <handshake>

The forwarder couldn't validate the authentication proof received from the indexer. Splunk software logs the failure as a WARN-level message and continues forwarding data in audit mode.

Search for outcomes of application-layer authentication

Use these searches to review application-layer authentication events, summarize S2S handshake outcomes, and monitor S2S authentication fleet metrics.

You can use the following searches to review application-layer authentication outcomes and metrics.

Objective Search syntax

Find application-layer authentication events logged by the TcpInputProc component on indexers.
CODE 
index=_internal sourcetype=splunkd component=TcpInputProc "S2S auth"

Summarize S2S authentication handshake attempts, successes, and failure types by host.
CODE 
index=_internal sourcetype=splunkd group=s2s_auth
| stats sum(handshake_failure) as handshake_failure,
        sum(invalid_proof) as invalid_proof,
        sum(no_common_epoch) as no_common_epoch,
        sum(handshake_timeout) as handshake_timeout,
        sum(strict_rejection) as strict_rejection,
        sum(downgrade) as downgrade,
        sum(handshake_success) as handshake_success,
        sum(handshake_attempts) as handshake_attempts
  by host

Review S2S authentication metrics across the deployment.

Search results include successful authentication handshakes and authentication-capable connections.
CODE 
index=_internal source=*metrics.log group=s2s_auth