Troubleshoot performance issues cause by searches and lookups in Splunk Enterprise Security

Troubleshoot performance issues cause by searches and lookups in Splunk Enterprise Security.

Issue

Performance issues or incomplete searches caused due to excessive memory usage by lookups or searches. You might see error messages such as Archiving large_file=(path/filename.csv) of size_in_bytes=#### (exceeding threshold=###). You might also see snapshot generation errors for knowledge bundle creation.

Causes

1. Indexing a search or a large lookup consumes excessive memory space: Indexing can impact performance as the size of the lookup grows larger. Smaller and denser lookups perform better in memory, while larger and sparser lookups perform better on disk. 25MB is the default for on-premises and 100MB is the default for cloud.

2. Lookup files are larger in size such as over 1GB: Lookup table files involved in special search matches, such as CIDR or Wildcard, are required to run in memory. This can lead to running out of memory when using these features.

3. Lookups do not follow the ASCII name order: Splunk Enterprise does not honor the lexicographical order of automatic search-time lookups when some of the lookups in a set are configured to run in-memory versus when some of the lookups in the set are configured to be indexed.

For instance, if you have max_memtable_bytes set to 50MB, assets_by_cidr lookup set to 25MB, and assets_by_str lookup set to 75MB. This can cause assets_by_str to be indexed and assets_by_cidr to run in memory, resulting in assets_by_cidr to inadvertently run prior to assets_by_str.

Solutions

1.Check the size of ../splunk/var/run/splunk/lookup_tmp file on the search heads: If the file is large and continuously increasing in size, some of the lookups might be corrupted. Control the maximum size of a lookup that can be indexed in memory by increasing the max_memtable_bytes in the $SPLUNK_HOME/etc/system/default/limits.conf configuration file. Use the following search to identify lookup files that are larger than the threshold:
CODE 
index=_internal sourcetype=splunkd "exceeding threshold"
            |dedup large_file
            |table large_file, size_in_bytes, threshold
Make the following changes to configuration settings in the limits.conf file:
1. Increase the value of the max_memtable_bytes = (integer) in the limits.conf configuration file:
Note:
  • Maximum size of the static lookup file in bytes that should be used as an in-memory index.
  • Lookup files with size above max_memtable_bytes are indexed on disk.
  • This setting max_memtable_bytes = (integer) also applies to lookup files that are loaded through the lookup() eval function that runs at search time. If the same function is called through the ingest-eval command, it uses the ingest_max_memtable_bytes setting instead.
  • Configuring this setting max_memtable_bytes = (integer) to a large value results in loading large lookup files in memory, which leads to a bigger process memory footprint. The default value is: 26214400 (25MB) and increasing this value impacts memory usage.
  • Though this setting max_memtable_bytes = (integer) is adjustable, you mustn't set the value equal to your biggest lookup without testing and tuning.

2. Increase the max_content_length setting: Increase the max_content_length of the http_input stanza in $SPLUNK_HOME/etc/system/default/server.conf.

When increasing httpServer:max_content_length in the server.conf configuration file, note that this setting exists to avoid allocating an unreasonable amount of memory from web requests.

Lookup table files that exceed the HTTP httpServer:max_content_length in the server.conf configuration file are not replicated across search head cluster members.

3. Configure the setting enforce_auto_lookup_order = true: Configure this setting in the [lookup] stanza of the limits.conf configuration file on the standalone search head or search peers and indexers so that the lookup names in the props.conf file are looked up in ASCII order by name.

This is the preferred method for the following Splunk Enterprise versions:

  • 8.1.5 and higher
  • 8.2.3 and higher
  • 9.0.0 and higher
  • 8.2.2106 and higher

See also

For more information on configuration files, see the product documentation:

  • limits.conf configuration file in the Splunk Enterprise Admin Manual.
  • server.conf configuration file in the Splunk Enterprise Admin Manual.