Support for tstats in searches on indexes with field filters

How to use the tstats command in searches on indexes that are protected by field filters.

Splunk platform supports searches that use search commands, as well as searches that use the tstats command on indexes protected by field filters. However, the behavior of tstats differs from other search commands in important ways because of the way this generating command accesses data.

See tstats in the Search Reference.

Field filter support for the PREFIX() directive

When you use the tstats command with the PREFIX() directive, make sure that the PREFIX arguments you select do not match or overlap with any sensitive fields listed in your field filters. This is because the Splunk platform protects sensitive data by blocking searches with the PREFIX() directive if either of the following is true:

  • The PREFIX argument starts with a sensitive field name.
  • A sensitive field name starts with the PREFIX argument.

Keep in mind that these searches are blocked on a per-index basis, only for the specific index where the field filter is applied.

For example, say you have a field filter for a sensitive field called password on the tstats_fieldfilter_local index, and you run the following search:

CODE 
| tstats values(PREFIX(passwo)) where index=tstats_fieldfilter_local

Your search is blocked because the prefix passwo starts with the password field that your field filter protects. When you change the prefix argument to another value, such as users in the following example, your search completes successfully:

CODE 
| tstats values(PREFIX(users)) where index=tstats_fieldfilter_local

Here are some additional example searches that are blocked or allowed depending on whether the prefix argument overlaps with field filters that protect a sensitive field called password on an index called tstats_fieldfilter_local:

Search Action Description
CODE 
| tstats values(password) where index=tstats_fieldfilter_local
 Allowed This search completes successfully because it doesn't include PREFIX().
CODE 
| tstats values(PREFIX(passwo)) where index=tstats_fieldfilter_local
CODE 
| tstats values(PREFIX(p)) where index=tstats_fieldfilter_local
 Blocked These searches don't complete successfully because they contain prefixes of password, and the tstats_fieldfilter_local index has a field filter for the sensitive field password.
CODE 
| tstats values(PREFIX(user)) where index=tstats_fieldfilter_local
 Allowed This search completes successfully because the argument user is not a prefix of password and password is not a prefix of user.
CODE 
| tstats values(PREFIX(password)) where index=user_info
 Allowed This search completes successfully because the search contains PREFIX(), but there aren't any field filters on the user_info index.
CODE 
| tstats values(PREFIX(password:)) where index= tstats_fieldfilter_local
CODE 
| tstats values(PREFIX(password=)) where index= tstats_fieldfilter_local
CODE 
| tstats values(PREFIX(password=123)) where index= tstats_fieldfilter_local
 Blocked These searches don't complete successfully because the sensitive field called password is a prefix of password:, password=, and password=123.
CODE 
| tstats values(PREFIX(passwordnew)) where index= tstats_fieldfilter_local
 Blocked

This search doesn't complete successfully because the sensitive field called password is a prefix of passwordnew.

Note: Even though passwordnew does not look like it should match the sensitive field password, users can define arbitrary delimiters, which could leak sensitive data. For example, if the delimiter is set to n, ne, or new, the password field still needs to be protected.

Only indexed field filters are applied in tstats searches

When you run searches with the tstats command, your searches operate only on indexed terms stored in tsidx files, not on _raw event data. This means that only indexed field filters are needed to protect sensitive fields in tstats searches, and _raw field filters are ignored.

Support for field filter limits differ in tstats searches

When you run tstats searches, field filters enforce limits on host, source, and source type where possible. However, in cases where these limits can’t be reliably enforced, field filters prioritize data protection by ignoring the limits or applying them in different ways, depending on which components are included in your tstats search.

The following table explains support for different field filter limits when tstats searches include one or more of the following components:
  • Aggregate functions, such as values, sum, and count, that perform statistical analysis on data.
  • The BY clause that groups by fields.
Limit field specified in the field filter Support for field filter limits in aggregate functions Support for field filter limits in the BY clause
host The field filter host limit is ignored and the field filter doesn’t apply the limit.

If the host field is explicitly included in the BY clause of the tstats search, the host limit is fully supported and the field filter applies the limit.

If the host field is not explicitly included in the BY clause of the tstats search, the field filter limit is ignored, and the given field filter is applied across all the values of host fields, instead of any specific values of the host field.
source The field filter source limit is ignored and the field filter doesn’t apply the limit.

If the source field is explicitly included in the BY clause of the tstats search, the source limit is fully supported and the field filter applies the limit.

If the source field is not explicitly included in the BY clause of the tstats search, the field filter limit is ignored, and the given field filter is applied across all the values of source fields, instead of any specific values of the source field.
source type The field filter sourcetype limit is ignored and the field filter doesn’t apply the limit.

If the sourcetype field is explicitly included in the BY clause of the tstats search, the sourcetype limit is fully supported and the field filter applies the limit.

If the sourcetype field is not explicitly included in the BY clause of the tstats search, the field filter limit is ignored, and the given field filter is applied across all the values of sourcetype fields, instead of any specific values of the sourcetype field.

For example, say you have a field filter on the _local index that redacts the ssn field, and sets a source limit for the sources called source1 and source2.

When you run the following search, the field filter will redact the value of the ssn field in events that originate from all sources in the index, not just source1 and source2.

CODE 
｜ tstats values(ssn) where index=user_ssn by user

However, if you run the following search, the field filter will only redact the value of the ssn field in events that originate from source1 and source2.

CODE 
｜ tstats values(ssn) where index=user_ssn by user, source

Field filters override fillnull_value in tstats searches

Field filters in tstats searches do not respect the fillnull_value argument, which replaces null values with a user-specified value for any field in the group-by list. When field filters are applied in tstats searches, they replace or delete sensitive fields and values depending on the field filter configuration, and ignore any fillnull_value settings in the search.

Field filters always delete fields and their values in certain tstats searches

Regardless of how the field filter replacement method is set, field filters always delete sensitive fields and their values when running searches with the tstats command in the following cases:

  • Field filters don't support field value replacement for searches that run the tstats command and the following aggregate functions on indexes protected by field filters:
    • avg(field)
    • min(field)
    • max(field)
    • range(field)
    • sum(field)
    • sumsq(field)
    • var(field)
    • varp(field)
    • stdev(field)
    • tdevp(field)

    Even if a field filter is set to hash or redact a field value, the field filter will instead delete the field and its value. Splunk software does this because these aggregate functions require numerical input, and these kinds of numeric functions should not be performed on sensitive fields protected by field filters. Deleting the field and its value ensures that functions on sensitive field data, such as sum(ssn) and sum(password), won’t appear in the aggregation output for these functions when used in tstats searches.

  • When searches run the tstats command with the prefix command in the format PREFIX(key=val) when the key represents a sensitive field. For example, in the following searches, the field filter deletes the result of PREFIX(ssn=12).
    CODE 
    | tstats count(PREFIX(ssn=12)) where index=main
    CODE 
    | tstats values(PREFIX(ssn=12)) where index=main

Field filter detection in tstats searches

Field filters in tstats searches detect sensitive fields in the where clause of your search after search filters have been applied. If a sensitive field appears anywhere in the final search, either because it is in your original tstats search or is introduced by a search filter , Splunk platform applies the corresponding field filter to fully protect the data.

Field filters apply their own limits on source, sourcetype, or host, independently of any scope specified in search filters. This design ensures comprehensive data protection because field filters are applied consistently whenever sensitive fields are present in the final search that runs, regardless of where those fields originate.

For example, say the following search filter is defined on the Restrictions tab for a new role called analyst:

NOT(source::/home/sensitive/data AND ssn::*)

In this example, the Splunk deployment also has a field filter called redact_ssn_with_string that replaces the value of the sensitive field ssn with the word redacted. The field filter is set on the tstats_fieldfilter index and is limited to the /home/app/log source.

Now let's say a user with the analyst role runs the following search:

CODE 
| tstats count where index=tstats_fieldfilter TERM(ssnhash=*)

The search filter adds the ssn field to the search, and the field filter is applied based on the final search that the Splunk platform runs:

CODE 
| tstats count where index=tstats_fieldfilter TERM(ssnhash=*)AND NOT(source=/home/sensitive/data AND ssn=*)

This search triggers the redact_ssn_with_string field filter because the final search contains the ssn field. Although the original search just referenced ssnhash and not ssn, the field filter is applied to the search according to its configured limits on the tstats_fieldfilter index and /home/app/log source. As a result, all ssnhash and ssn fields originating from those indexes and sources are protected when the analyst runs their search.