Fields and field values

Note: Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

Before you begin, see Plan for field filters in your organization for important considerations about planning for field filters.

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters are a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone.

If your organization uses downstream configurations, such as accelerated data models, Splunk Enterprise Security (ES) detections using those data models, and user-level search-time field extractions, make sure that you plan around the implications of field filters on those configurations before deploying field filters in your environment. See READ THIS: Downstream impact of field filters.

If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview and mstats), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes that have field filters.

Tracking fields you need to secure

Which sensitive data in your events do you need to protect? Are there certain indexed fields that contain personal identifiable information (PII), protected health information (PHI), or other confidential data that you need to keep private? You might consider creating a document that helps you keep track in one place of all the fields with values that you need to secure, along with descriptions of their field filters.

Replacement methods

Before you create your field filters, plan your field filters strategy and identify the specific fields that need to be deleted from the search results, or replaced with a custom string or hash. Your approach depends on the level of protection your data needs.

  • If your data needs the highest form of privacy for legal or regulatory reasons, your plan might need to include redacting field values by deleting them or replacing them with a custom string. The delete option in Splunk Web removes the field and the custom string option replaces the field with a string value.

  • If you plan to obfuscate your field values with SHA-256 or SHA-512 (SHA-2 family) hash functions, make sure that replacing field values with a hash offers the security your data requires.

  • If you want to preserve the uniqueness of a field for statistical use cases, for example, when using the stats or timechart commands in searches, plan to replace the field value with a hash.

Indexed and _raw field types

Field filters can be applied to fields that are extracted at index time or later at search time, which ensures that sensitive fields are protected regardless of when they're processed. As a result, Splunk platform supports two types of field filters:

  • Indexed field filters, which are used to filter default fields that the Splunk platform automatically adds to the index, and any custom fields that you define.

  • _raw field filters, which are used to filter fields extracted from event data at search time, typically from _raw fields, which are the source events that appear in search results. A _raw field is a type of default field that contains the original source data of events and is typically used as the basis for index-time and search-time field extractions.

To keep confidential data secure, it is important to apply field filters to indexed fields and the corresponding _raw fields since your _raw fields contain the same confidential information that your indexed field filters safeguard. If you apply a field filter to an indexed field, but don't also apply a field filter for the _raw field that matches the events from which the indexed field was extracted, users without authorization to view confidential information in the indexed field might still be able to access that information through the _raw field. To keep all of your confidential data safe and ensure that your indexed field filters can't be circumvented, create a _raw field filter that uses regular expressions to filter on each _raw field that extracts fields that your indexed field filters protect.

A single index can have multiple field filters on _raw and individual indexed fields:

  • Each indexed field can have only one field filter per index.

  • Each _raw field can have multiple _raw field filters per index. The _raw field filters are run in lexicographical order of their names.

Identify index-time fields

Typically, most of your field filters will be for _raw fields; you only need to create indexed field filters when a field is indexed. To determine whether the field you want to protect needs an indexed field filter, you need to identify if the field is indexed or extracted at search time. Sometimes it can be difficult determine when a field is extracted because Splunk Web doesn't explicitly label fields with their extraction type in search results. Typically, fields such as host, source, sourcetype, index, and _time, and custom fields that you define, are automatically added to the index at index-time, and need indexed field filters. Any other fields are search-time fields extracted from _raw data, and need _raw field filters.

To identify an index-time field, you can run a search like this in Splunk Web:

CODE 
| tstats values(component) where index=main

If the search returns results, then the component field exists in the _main index. But, if the search doesn't return any results, then component is not an indexed field.

Alternatively, you can run a search like this in Splunk Web, which returns a list of field names that are indexed in the _main index:

CODE 
|walklex index=main type=field

You can use this list of fields to decide which indexed field filters you need to create for the _main index. See walklex.

Supported delimiters for indexed terms in tstats, walklex, and typeahead searches with field filters

In searches that use the tstats, walklex, or typeahead commands on indexes protected by field filters, key-value pairs for term and field values in raw events must be in one of the following formats, in order to be protected by field filters:
  • <field>=<value>
  • <field>::<value>

Key-value pairs that use other delimiters, such as dollar sign ( $ ), colon ( : ) or comma ( , ), are not protected by field filters. To ensure your field filters work correctly, make sure your raw data uses either double colon ( :: ) or equal sign ( = ) between fields and values.

Impact of indexed fields on downstream operations

Another factor to consider is the implications of using indexed fields for downstream operations that depend on the value of the field that is changed by a field filter. See The sequence of search-time operations ​in the Knowledge Manager Manual.

Next step

Next, plan how to name your field filters and how many you need. See Field filter naming and quantity.