What does the Catalog show me?

The Catalog in Splunk Cloud Platform shows you the datasets and Machine Data Lake promotion jobs that you can access.

The Catalog is organized into 2 tabs:

  • The Datasets tab displays the datasets that are available to you based on the roles granted to your user account and the permissions that are configured in each dataset. For more information, see Datasets.

  • The Promotion jobs tab displays any Machine Data Lake promotion jobs that are available to you. For more information, see Promotion jobs.

Datasets

Browse datasets and inspect dataset metadata using the the Datasets tab of the Catalog in Splunk Cloud Platform.

The Datasets tab of the Catalog supports the following types of datasets:

Dataset type Description
Analytics table

Contains a subset of promoted data from a raw table, and is optimized for structured analysis over a selected historical time range and selected fields.
Federated dataset

Contains searchable data from a remote source, and in some cases can be used as destinations for Edge Processor and Ingest Processor pipelines.

Federated datasets include the following kinds of datasets from the Data Management app:

  • Amazon S3

  • Azure Databricks

  • Dynamic Data Self Storage (DDSS)

  • Microsoft Azure

  • Snowflake
Raw table

Contains data that lands in Machine Data Lake with only minimal processing, such as the following:

  • Line breaking

  • Timestamp assignment

  • Extraction of the source, sourcetype, host, and _time fields

You can promote specific subsets of data from raw tables to analytics tables or Splunk indexes for fuller search and analysis functionality.
Splunk index

Contains data in the Splunk platform deployment that is processed for search optimization.

You can create indexes independently or by promoting a subset of data from a raw table.

Splunk indexes include both event indexes and metric indexes.

Viewing details about datasets

You can select a dataset on the Catalog to view more information:

  • The resource name of the dataset

  • The timestamps of the earliest and latest events in the dataset

  • The number of events in the dataset

  • The type of data source that the events originated from

  • The associated promotion jobs and the type of promotion used to populate the dataset

The information in the side panel varies depending on the dataset type and whether optional features such as Schema Collection or MDL Raw Dataset Statistics are turned on.

For example, dataset statistics like the number of events or the earliest and latest event timestamps have the following differences in availability:

  • Available for Splunk indexes by default

  • Available for raw tables and promoted datasets only when the MDL Raw Dataset Statistics feature is turned on

  • Not available for federated datasets

As another example, information about promotions is exclusive to raw tables, analytics tables, and Splunk indexes that are the result of a promotion job.

Note: The list of datasets on the Catalog and the dataset statistics in the side panel are subject to a 30-minute refresh rate. Newly added datasets and changes to the dataset statistics are not reflected immediately in the Catalog.

For information about the optional features that Splunk Cloud Platform administrators can choose to activate for the Datasets tab of the Catalog, see the following sections:

Inspecting event fields in datasets

By default, the Catalog does not display detailed information about the events inside the datasets. However, Splunk platform administrators can configure the Catalog to show event field names in each dataset by turning on a background search operation called Schema Collection. When Schema Collection is turned on, the dataset side panels show a Fields section that lists the names of the available event fields.

Inspect these field names to learn about the data schema and start your investigations with more targeted searches on the dataset, eliminating the need for broad, exploratory SPL or SPL2 searches that can be costly and resource-intensive.

For more information about how an administrator can turn on Schema Collection, see Configure background search operations for the Catalog in the Splunk Cloud Platform Admin Manual.

Note:

The event field information is subject to the following limitations:

  • Event fields are not available for metrics indexes or federated datasets where the schema is defined in a customer-managed data catalog such as AWS Glue or Apache Iceberg REST.

  • The list of field names for newly added datasets is populated in phases. When the Catalog retrieves the field names for a dataset for the first time, it runs a search every 60 minutes until all the information is retrieved.

  • The list of field names for existing datasets is subject to a daily refresh rate. Updates to the field names available in the dataset are not reflected immediately in the Catalog.

Viewing dataset statistics for raw tables and promoted datasets

By default, the Catalog does not display information about event time ranges and sizes for raw tables, analytics tables, and promoted Splunk indexes. However, Splunk platform administrators can configure the Catalog to show this information by turning on a background search operation called MDL Raw Data Statistics. When MDL Raw Data Statistics is turned on, you can view the following information in the Data summary section of the side panel for raw tables and promoted datasets:

  • Event count

  • Earliest event

  • Latest event

  • Size

Browse this information to learn about the scope of the data in the selected dataset and identify whether it is relevant to your investigation.

For more information about how an administrator can turn on MDL Raw Data Statistics, see Configure background search operations for the Catalog in the Splunk Cloud Platform Admin Manual.

Note: The dataset statistics in the side panel are subject to a 30-minute refresh rate. Changes to the dataset statistics are not reflected immediately in the Catalog.

Promotion jobs

Manage Machine Data Lake promotion jobs using the Promotion jobs tab of the Catalog in Splunk Cloud Platform.

Use promotion jobs to process subsets of data from Machine Data Lake raw tables into datasets that are more optimized for searches. The Promotion jobs tab of the Catalog displays information about the promotion jobs that you own or have permission to manage, such as the following:

  • The name of the dataset created by the promotion job

  • Whether the promoted dataset is an analytics table or Splunk index

  • Whether the dataset was populated by a static promotion or streaming promotion

  • The status of the promotion job, such as Active, Complete, or Cancelled

  • The date and time when the promotion job was created

Select a promotion job to view additional details such as when the status was last updated and the raw table that the promoted data originates from.

For complete information about promotions, see the Machine Data Lake manual.