Troubleshoot Splunk forwarder TCP tokens

You can control which forwarders in your Splunk Enterprise deployment have access to the indexers by setting up forwarder TCP tokens. In the case where a forwarder TCP token becomes corrupt, or the indexer to which the forwarder sends data rejects the token, that indexer generates an error message in its logs. A forwarder always tries to communicate with the indexer you have configured it to communicate with, regardless of whether or not the TCP token you have configured for it is valid.

To locate the bad forwarder token, increase the logging level of the indexer:

  1. Open a shell prompt.
  2. Using a text editor, edit SPLUNK_HOME/etc/log.cfg as follows: 
    category.TcpOutputProc=DEBUG
category.TcpInputConfig=DEBUG
category.TcpInputProc=DEBUG
  3. Save the file and close it.
  4. Restart the indexer.

When a token that a forwarder sends matches the token that the indexer receives, Splunk components generate the following messages:

Indexer: 

09-15-2015 13:21:30.746 -0700 DEBUG TcpInputProc -  Forwarder token matched

Universal Forwarder: 

09-15-2015 13:24:00.343 -0700 DEBUG TcpOutputProc - Indexer 
can use tokens

When the tokens do not match, the indexer generates a message similar to the following: 

09-15-2015 13:22:01.747 -0700 ERROR TcpInputProc - Exception:  Token not sent by 
forwarder src=10.140.126.58:51838! for data received from src=10.140.126.58:51838

09-15-2015 13:52:14.803 -0700 ERROR TcpInputProc - Exception:  Token sent by 
forwarder does not match configured tokens src=10.140.126.58:51990! 
for data received from src=10.140.126.58:51990