Datasets
Browse datasets and inspect dataset metadata using the Datasets tab of the Catalog in Splunk Cloud Platform.
The Datasets tab of the Catalog displays the datasets that are available to you based on the roles granted to your user account and the permissions that are configured for each dataset. In Splunk Cloud Platform deployments with search head clusters, the Catalog lists analytics tables, federated datasets, and Splunk indexes created through promotions only in the specific search head where the dataset or promotion job was created. Raw tables and non-promoted Splunk indexes are available across all the search heads in the cluster.
Use datasets to store, search, and analyze data in Splunk Cloud Platform and remote data stores. For example, you can search a Splunk index for relevant events and use those events to build reports and dashboards. As another example, you can send processed data from Edge Processor and Ingest Processor pipelines to Amazon S3 buckets through a federated dataset.
You can use the Datasets tab to view and manage the following types of datasets:
| Dataset type | Description |
|---|---|
| Analytics table |
Contains a subset of promoted data from a raw table, and is optimized for structured analysis over a selected historical time range and selected fields. |
| Federated dataset |
Contains searchable data from a remote source, and in some cases can be used as destinations for Edge Processor and Ingest Processor pipelines. Federated datasets include the following kinds of datasets:
|
| Raw table |
Contains data that lands in Machine Data Lake with only minimal processing, such as the following:
You can promote specific subsets of data from raw tables to analytics tables or Splunk indexes for fuller search and analysis functionality. |
| Splunk index |
Contains data in the Splunk platform deployment that is processed for search optimization. You can create indexes independently or by promoting a subset of data from a raw table. Splunk indexes include both event indexes and metric indexes. |
The Datasets tab displays information such as the type, owner, creation date, and status of each dataset. You can immediately use any datasets in the Ready status, but datasets in other statuses might require more time for internal processes to complete or an administrator to correct the configuration settings before you can use them.
Viewing details about datasets
You can select a dataset on the Catalog to open a side panel and view details such as:
-
The resource name of the dataset
-
The timestamps of the earliest and latest events in the dataset
-
The number of events in the dataset
-
The type of data source that the events originated from
-
The name of the connection that the dataset uses to access a remote data store
-
The associated promotion jobs and the type of promotion used to populate the dataset
The information in the side panel varies depending on the dataset type and whether optional features such as Schema Collection or MDL Raw Dataset Statistics are turned on.
For example, dataset statistics like the number of events or the earliest and latest event timestamps have the following differences in availability:
-
Available for Splunk indexes by default
-
Available for raw tables and promoted datasets only when the MDL Raw Dataset Statistics feature is turned on
-
Not available for federated datasets
As another example, information about promotions is exclusive to raw tables, analytics tables, and Splunk indexes that are the result of a promotion job.
For information about the optional features that Splunk Cloud Platform administrators can choose to activate for the Datasets tab of the Catalog, see the following sections:
Inspecting event fields in datasets
By default, the Catalog does not display detailed information about the events inside the datasets. However, Splunk platform administrators can configure the Catalog to show event field names in each dataset by turning on a background search operation called Schema Collection. When Schema Collection is turned on, the dataset side panels show a Fields section that lists the names of the available event fields.
Inspect these field names to learn about the data schema and start your investigations with more targeted searches on the dataset, eliminating the need for broad, exploratory SPL or SPL2 searches that can be costly and resource-intensive.
For more information about how an administrator can turn on Schema Collection, see Configure background search operations for the Catalog in the Splunk Cloud Platform Admin Manual.
The event field information is subject to the following limitations:
-
Event fields are not available for metrics indexes or federated datasets where the schema is defined in a customer-managed data catalog such as AWS Glue or Apache Iceberg REST.
-
The list of field names for newly added datasets is populated in phases. When the Catalog retrieves the field names for a dataset for the first time, it runs a search every 60 minutes until all the information is retrieved.
-
The list of field names for existing datasets is subject to a daily refresh rate. Updates to the field names available in the dataset are not reflected immediately in the Catalog.
Viewing dataset statistics for raw tables and promoted datasets
By default, the Catalog does not display information about event time ranges and sizes for raw tables, analytics tables, and promoted Splunk indexes. However, Splunk platform administrators can configure the Catalog to show this information by turning on a background search operation called MDL Raw Data Statistics. When MDL Raw Data Statistics is turned on, you can view the following information in the Data summary section of the side panel for raw tables and promoted datasets:
-
Event count
-
Earliest event
-
Latest event
-
Size
Browse this information to learn about the scope of the data in the selected dataset and identify whether it is relevant to your investigation.
For more information about how an administrator can turn on MDL Raw Data Statistics, see Configure background search operations for the Catalog in the Splunk Cloud Platform Admin Manual.