Improve search performance with quick searches in Splunk Enterprise Security 8.8 and higher

When you upgrade to Splunk Enterprise Security version 8.8 and higher, quick searches are run by default instead of Splunk searches on the analyst queue to improve search time performance.

Quick searches read the mc_incidents KV Store collection directly and do not run a Splunk search. The search session record is short-lived. To run quick searches, you must have the ingest_default_queue_findings setting in the [optimization] stanza of the mc_database.conf file set to true. The default value is true). However, if you have it turned off, findings on the queue won't be displayed.

Quick search on the Analyst queue supports sorting and advanced filters but is not available for the following scenarios:

  • Sort on the analyst queue using non KV compatible fields such as Entity risk score and Owner.
  • Search using the search bar.
  • Search that includes the contains or not contains advanced filters.
  • Time range is older than the queue's data retention period.

When you sort investigations using the Title field, investigations without child findings won't be sorted but are displayed in the beginning or the end of the sort results in the analyst queue.

The analyst queue also falls back to a Splunk search if there is no queue filter, suppressed findings, or raw event filters. Additionally, filtering or sorting on Severity, rule name, finding type, related investigations, risk score or owner name, and agent response progress causes a reversion to Splunk search.

A queue that is set to delete immediately with a retention period of 0 never uses quick search and the default queue skips the retention check.

Findings and investigations that are removed due to the retention policy are hidden from the queue. You can use the /unarchive REST API endpoint to unarchive an investigation. For more information on the /unarchive REST endpoint, see the Splunk Enterprise Security API Reference manual.

If you search for findings that are part of an investigation in Splunk Enterprise Security, the parent investigation is also identified. However, if you search for findings that are part of an investigation, which was created in versions prior to Splunk Enterprise Security 8.8, the investigation is not displayed.

Similarly, If you search for custom fields that are part of an investigation in Splunk Enterprise Security, the parent investigation is also identified. However, when you search for custom fields in the analyst queue that are part of investigations created in versions prior to Splunk Enterprise Security 8.8, the investigations are not displayed.

When you sort investigations using the Title field, investigations that don't include child findings won't be sorted but are displayed in the beginning or the end of the sort results in the analyst queue.

Toggle quick search in general settings

Follow these steps to turn on quick search for the analyst queue in the general settings of Splunk Enterprise Security version 8.8 and higher:
  1. In Splunk Enterprise Security, go to Configure.
  2. Select General and then select System settings.
  3. Enter mission control in the Apps search bar to access the configuration files.
  4. Select mc_search.conf configuration file.
  5. Go to the [aq_sid_caching] stanza and turn on the quick_search setting.
    Note: In Splunk Enterprise Security version 8.8 and higher, the quick_search setting is turned on by default.
    Turning on the quick search flag lets the analyst queue to query the KV Store collection directly (instead of using Splunk search) when the filters match supported fields and the time range is within the KV Store retention window.
  6. In the [aq_sid_caching] stanza, then go to the quick_search_max_ttl setting and enter a value. The maximum time that the quick search's KVStore record is kept active in the KV store is 20 minutes by default. This value resets every time the session is used. If the search metadata record isn't refreshed or accessed within 20 minutes, it is wiped from the KV store to preserve memory and improve performance.
  7. In the [aq_sid_caching] stanza, go to the quick_search_stats_batch_size setting and enter a value to specify the number of KV store records fetched for each page by the quick search. The default value for this setting is 10001. The worker also fetches 5 pages concurrently, so lowering this value to 500 still means 5×500-record requests in parallel. Lowering the batch size of the processing load to 500 can help ease the possibility of the KV store timing out and reduce the memory load on your search head.
  8. In the [aq_sid_caching] stanza, go to the max_quick_search_events setting to specify the maximum number of KV records that a quick search scans to compute results. This setting has a default value of 10000. This setting caps the result count, field statistics, timeline, and bulk-action selection.
Note: Configuration changes can take up to 10 minutes to apply.