Identify the time field in an Azure Databricks dataset

Configure time settings for your Azure Databricks dataset to use time-based fields and functions in federated searches.

Note: In the Controlled Availability release stage, Splunk products may have limitations on customer access, features, maturity, and regional availability. For additional information on Controlled Availability please contact your Splunk representative.

If your Azure Databricks dataset contains a time field and you want to run searches over the dataset that involve time-based filters and functions, you must identify this time field in the dataset definition. Splunk software cannot identify a remote dataset's time field for you.

When you specify the time field, you must also supply the format for the time field, and identify a Unix time field value, such as _time, that Splunk software can convert into numeric UNIX time format at search time.

For more information about these time settings, see the following definitions.

Time field

Enter the name of the field that acts as an event timestamp in the Azure Databricks dataset.

The time field can contain only lowercase letters, numbers, underscores, and dot characters (.).

Surround time fields that contain dot characters, but which are not nested fields, with single quote characters.

Time format

Provide a time format variable or custom time format variable string that matches the Time field.

  • Set %s when the Time field has UNIX time values with the string data type.
  • Set %UT when the Time field has UNIX time values with the numeric data type.
  • Set %ST when the Time field has values with the SQL timestamp data type.
  • Set a custom string of time format variables when the Time field has values that follow a specific string time format. For more information, see Using time variables in the SPL2 Search Manual.
Note: %UT and %ST are not among the standard set of Splunk platform time format variables. Use them only in the context of Federated Search for Azure Databricks.

You can optionally append the %Q time format variable to capture subsecond timestamps, such as milliseconds (%3Q), microseconds (%6Q), and nanoseconds (%9Q).

Unix time field

The Unix time field provides an alias for the Time field that Splunk software converts into numeric UNIX time format at search time. Insert the Unix time field into federated searches that require numeric UNIX time field values, or when you want to see your time field in numeric UNIX time format in the search results.

The Unix time field defaults to _time. In Splunk Web, the values of _time always display in human-readable format, unless you are aggregating on the _time field. For example, (avg)_time returns values in numeric UNIX time format.

Note: If _time already exists as a field name in your dataset schema, give the Unix time field a value other than _time.