Flexible data typing in SPL2

SPL2 provides an extensible data typing system that you can use to define the schema of your data on an as-needed basis:

• You can control when to constrain data to specific types and when to leave the data loosely typed.

• You can choose to constrain various levels of data to different types. For example, a dataset can contain records that match the object type, and the individual fields in the records can match other types such as string or number.

• You can expand beyond the default set of supported data types by creating fine-tuned custom data types that match the shape of your actual data.

You can also choose to not use data types, and instead rely on the loose typing logic that the Splunk platform uses by default.

When to use strong typing instead of loose typing

SPL2 is loosely typed by default. You don't need to define specific types for your data before successfully ingesting, searching, and processing it, and you can define functions that allow the input and output to be any type of data. This flexibility reduces the amount of overhead required to work with the language, and supports use cases where the schema of the data is unknown or highly variable.

However, there are also situations where it is beneficial to constrain your data to specific types. Compared to strongly typed data, loosely typed data is relatively ambiguous. For example, if an event field named customer is untyped, it is unclear if the field should contain names, ID numbers, detailed records in JSON object format, or something else. The field might even contain a mix of those values if consistency is not enforced during data entry.

You can use strong typing to eliminate this ambiguity and set logical guidelines for your data. For example, assume that the customer event field is intended to contain ID numbers only. You can make this requirement explicit by constraining the customer field to the int data type, which corresponds to integers or whole numbers.

• You can verify whether all the values in the customer field are ID numbers instead of names or other kinds of text strings. The following expression returns true if the customer value matches the int type, and returns false otherwise:

  ... | eval type_check_results = if(isint(customer), "true", "false")

• You can filter the data so that only records that have ID numbers in the customer field are allowed to continue downstream. The following expression filters the data and only retains records where the value in the customer field matches the int type:

  ... | where isint(customer)

Specifying precise data requirements using custom data types

In addition to using the default data types that are built into SPL2, you can also create custom data types that specify more precise requirements.

For example, assume that a valid customer ID number must be exactly 6 digits, and that the first digit cannot be a 0. In this case, constraining the customer field to the int type will not suffice, since integers include numbers that have fewer or more than 6 digits as well as numbers that have 0 as the first digit. To capture these requirements for customer ID numbers, you can define a custom data type that is based on the int type but restricts the allowed range of values. The following expression defines a custom type named id_number, which corresponds to integers that are between 100000 and 999999, inclusive:

type id_number = int where ($value BETWEEN 100000 AND 999999)

You can then constrain the customer field to this custom type.

• The following expression returns true if the customer value matches the id_number type, and returns false otherwise:

  ... | eval type_check_results = if(customer IS id_number, "true", "false")

• The following expression filters the data and only retains records where the value in the customer field matches the id_number type:

  ... | where customer IS id_number

You can also combine multiple data types together in order to create an advanced data type that layers together the requirements defined in those individual types.

As another example, assume that the _raw field in your data contains complete customer records, which are JSON objects containing the keys customer, name, email, and vip_member. A valid customer record looks like this:

{customer: 109351, name: "Buttercup Games Company", email: "info@buttercupgames.com", vip_member: "true"}

You can use a combination of data types to constrain the values in the _raw field to this customer record format and ensure that the values for each key are valid. To do this, start by identifying data types that describe the validation requirement for each key in the JSON object:

type id_number = int where ($value BETWEEN 100000 AND 999999)

type email_address = string WHERE match($value, /(?P<Email>[a-zA-Z][a-zA-Z0-9._-]+@(\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)))/);

Then, to specify how these data types must be contained in a JSON object format, combine them in a structured data type named customer_record:

type customer_record = {
    customer: id_number,
    name: string,
    email: email_address,
    vip_member: boolean
}

You can then constrain the _raw field to this customer_record type in order to ensure that the customer records are all JSON objects that contain the keys customer, name, email, and vip_member, and that the value in each key is valid according to your requirements.

By constraining a piece of data to a particular type, you can specify and enforce rules about the structure and content of that data.

Using schemas to work with your data

When you create a schema for your data using data types, you implement a metadata framework that describes the characteristics of the data, such as the relevant fields and the allowed range of values. This framework allows you to distinguish between different kinds of data, identify data of interest, and selectively filter and process specific subsets of data.

Depending on your particular needs, you might want to use schemas during ingest time, search time, or both. Constraining incoming data to types as it streams through an Edge Processor or Ingest Processor pipeline allows you to categorize that data, process and route different categories differently, and validate and correct the data before writing it to storage. In contrast, constraining search results to types allows you to categorize and selectively process data while investigating, analyzing, and reporting on indexed data.

Across all contexts, schematizing your data allows you to achieve a broad range of data processing outcomes, including the following:

See also