Reduce system load by managing and retrieving search result caches in the Splunk App for Data Science and Deep Learning

When working with compute-intensive algorithms like large language models (LLMs), it is crucial to efficiently manage and retrieve search results in order to reduce system load and ensure timely access to valuable insights.

The Splunk App for Data Science and Deep Learning (DSDL) version 5.2.1 introduce Search History that uses the Splunk platform built-in summary index to persist and reuse expensive search results.

Benefits of using search history

Review the following table for benefits of the search history feature:

Benefit Description
Persist valuable outputs By default, search results in DSDL are not stored. If you need a result again, you must rerun the search. Using search history saves search results for future access.
Performance and efficiency Storing results in a summary index means you can quickly retrieve previously processed data, reducing the need to repeat resource-heavy computations. This is especially beneficial for algorithms that return lengthy text outputs or when auditing past analyses.
Reuses standard Splunk platform features By using the Splunk platform built-in summary index, the feature ensures compatibility, reliability, and security for your stored search data.

Turn on search history

To access previously cached search results in the Splunk App for DSDL, complete the following steps:

CAUTION: Turning on history for algorithms that output large numeric tables might lead to significant performance or storage concerns. Consider selectively enabling this feature only where high resource consumption or frequent result reuse is expected.
  1. Turn on Essential Saved Searches in your Splunk platform instance. These settings are stored in the saved searches settings:
    • get_search_jobs_save_to_summary
    • get_search_results_save_to_summary
  2. Select the algorithms you want to toggle on for history retention:
    1. Edit the search_history_enabled_algos.csv lookup table containing both algo_name and algo_enabled.
    2. Find the name of the algorithm you want to enable and set its value in the algo_enabled column to 1.

      Note: By default, only LLM algorithms have history enabled, because they often involve lengthy processing times and generate extensive text outputs.

After you turn on search history, the following changes take effect:

  • Results for selected algorithms are stored in summary indexes.
  • Users can retrieve these results instantly for quick analysis or auditing, without the need to rerun the original, potentially expensive, search.