Configure background search operations for the Catalog

Configure background Catalog search operations in Splunk Cloud Platform to prevent Catalog information from becoming stale or incomplete.

The Catalog is a page that displays information about the datasets and Machine Data Lake (MDL) promotion jobs that are available to a given user, based on the roles granted to the user and the permissions configured in the datasets. To retrieve information and populate the Catalog, Splunk Cloud Platform periodically runs search operations in the background.

If necessary, you can turn these background search operations on or off by submitting a request to Splunk customer support.

Before requesting any configuration changes, review this page to learn more about the purpose and impact of each search operation.

Search operations

The following table summarizes the background search operations that support the Catalog:

Operation name Purpose Usage and impact Search frequency Counts towards the concurrent search limit?
Splunk Index Discovery Retrieves Splunk indexes and includes information such as name, type, and dataset statistics
Note: Splunk Index Discovery is required to support core, critical functionality in MDL and the Catalog. Turning off this operation causes MDL and the Catalog to stop working.

Splunk Index Discovery is turned on by default.

This operation is required for the Catalog's core feature of discovering and listing Splunk indexes. It does not use SPL or SPL2 searches to retrieve index information, and so it has negligible impact on system performance and SVC consumption.

 Every 30 minutes No
MDL Metadata Field Values Retrieves the values of the default event fields (source, sourcetype, host) in raw tables and promoted datasets
Note: MDL Metadata Field Values is required to support core, critical functionality in MDL and the Catalog. Turning off this operation causes MDL and the Catalog to stop working.

MDL Metadata Field Values is turned on by default.

This operation is required for core features of Machine Data Lake and the Catalog, including the following:

  • Filtering datasets by field-value pairs and event time range

  • Promotion planning, which involves source type discovery and data size estimation

This operation can have low to medium impact on search concurrency and SVC consumption.

 Every 30 minutes Yes
Schema Collection Retrieves the list of index-time extracted fields for Splunk indexes

Schema Collection is turned off by default.

This optional operation allows the Catalog to display the Fields section in the side panel that opens when you select a Splunk index. The Fields section shows the names of the event fields in the selected index.

To allow users to inspect the schema of their Splunk indexes before they run any SPL or SPL2 searches, turn on Schema Collection.

This operation can have low to medium impact on search concurrency and SVC consumption, depending on the amount of data indexed in the Splunk platform deployment.

  • When retrieving information about a newly created index for the first time: Every 60 minutes until all the information is retrieved.

  • When retrieving updates for an existing index: Daily

 Yes
MDL Raw Dataset Statistics Retrieves information about the time ranges and sizes of events in raw tables, analytics tables, and promoted Splunk indexes

MDL Raw Dataset Statistics is turned off by default.

This optional operation allows the Catalog to display the following information in the Data summary section of the side panel that opens when you select a raw table or promoted dataset:

  • Event count

  • Earliest event

  • Latest event

  • Size

To allow users to learn more about the scope of the data in their raw tables and promoted datasets before they run any SPL or SPL2 searches, turn on MDL Raw Dataset Statistics.

This operation can have low to medium impact on search concurrency and SVC consumption.

 Every 30 minutes Yes

See also