Troubleshoot missing agents

The number of agents displayed on the interface is less than the actual number of connected agents, or saved search results show errors and messages related to truncation.

You observe one of the following symptoms:

  • The number of agents displayed on the interface is 10,000, but the actual number of connected agents is higher.

  • The number of agents displayed on the interface is 50,000, but the actual number of connected agents is higher.

  • When you check the saved search results, error messages related to result truncation are displayed.
The default limits of the following settings are too low, causing results to be truncated:

  • *[subsearch] `maxout` - default 10000

  • *[searchresults] `maxresultrows` - default 50000

  1. Open the limits.conf file that is located in $SPLUNK_HOME/etc/system/local/limits.conf.
  2. Increase maxout setting in the configuration. Under the [subsearch] stanza, set the following setting: maxout = 75000 * Default = 50000. Set a value that accommodates your expected maximum number of agents.
  3. If you set the value of maxout above 50000, you need to increase two more settings in the limits.conf file:
    1. Increase the maxresultrows setting under the [searchresults] stanza: 
      maxresultrows = 75000
* Default = 50000

      Set a value that accommodates your expected maximum number of agents.

    2. Increase the subsearch_maxout setting under the [join] stanza: 
      subsearch_maxout = 75000
* Default = 50000

      Set a value that accommodates your expected maximum number of agents.

  4. Save your changes.
  5. Restart your Splunk instance for the change to take effect.