Identify time partitions

Identify time partition fields in your dataset definition to improve federated search performance and reduce search cost.

Partitioning is an organization strategy for large datasets that enables you to search them efficiently. When you partition your data, you organize it into a hierarchical directory structure based on the distinct values of one or more fields in the data.

For example, you might partition your application logs in Amazon S3 by date, breaking them down by year, month, and day. Then you can place files corresponding to a single day's worth of data in an Amazon S3 path like s3://my_bucket/logs/year=2025/month=08/day=23/ if your dataset uses Hive-style partitions or s3://my_bucket/logs/2025/08/23/ if your dataset does not use Hive-style partitions.

Note: You can identify time partition settings for a dataset even if you have not selected Define the time field. Time partition fields exist in the Amazon S3 paths that form the structure of your dataset. The Time field exists as a column in the data catalog that references your dataset.

When you define time partitions for a dataset, identify the first field in the time field hierarchy, then the second field, and so on. For example, if your data catalog references a dataset that you have partitioned by year, month, and day, identify year as the first time partition field, month as the second time partition field, and day as the third time partition field.

Define time partition fields

  1. In your dataset definition, under Time partition settings, select the Time zone that applies to your time partition fields. You must choose a time zone if you define one or more time partition levels.
  2. Select Add field.
  3. Identify the first field by which your dataset is partitioned. Specify values for the following settings:
    Time partition setting Description
    Time partition field Provide the name of the time field that is the partition key for the indicated partition filter level. Values can contain only lowercase letters, numbers, and underscores.
    Time format Provide a time format string for the indicated Time partition field. Compose this time format string out of Splunk-supported time format variables. For more information, see Using time variables in the SPL2 Search Manual.
    Data type Select the data type of the Time partition field. Your options are String, Integer, and Date.
  4. If you have another time partition key, select Add field and identify the Time partition field, Time format, and Data type. Repeat until you have defined a partition level for each partition key you want to use in your federated searches.