Support for tstats in searches on indexes with field filters

Splunk platform supports searches that use search commands, as well as searches that use the tstats command on indexes protected by field filters. However, the behavior of tstats differs from other search commands in important ways because of the way this generating command accesses data.

See tstats in the Search Reference.

Supported delimiters for indexed terms in tstats searches with field filters

To keep your sensitive data in tstats searches safe, field filters overprotect, rather than underprotect, your data by filtering both indexed terms and indexed fields, provided that they are in one of the following formats:
  • <field>=<value>
  • <field>::<value>

You can use field filters on indexed fields with the tstats command to filter indexed terms that use the format <field>=<value>, and indexed fields that use the format <field>::<value>.

Field filters in tstats searches do not protect terms and fields in key-value pairs from original source events that use other delimiters, such as dollar sign ( $ ), colon ( : ) or comma ( , ).

Only indexed field filters are applied in tstats searches

When you run searches with the tstats command, your searches operate only on indexed terms stored in tsidx files, not on _raw event data. This means that only indexed field filters are needed to protect sensitive fields in tstats searches, and _raw field filters are ignored.

Support for field filter limits differ in tstats searches

When you run tstats searches, field filters enforce limits on host, source, and source type where possible. However, in cases where these limits can’t be reliably enforced, field filters prioritize data protection by ignoring the limits or applying them in different ways, depending on which components are included in your tstats search.

The following table explains support for different field filter limits when tstats searches include one or more of the following components:
  • Aggregate functions, such as values, sum, and count, that perform statistical analysis on data.
  • The BY clause that groups by fields.
Limit field specified in the field filter Support for field filter limits in aggregate functions Support for field filter limits in the BY clause
host The field filter host limit is ignored and the field filter doesn’t apply the limit.

If the host field is explicitly included in the BY clause of the tstats search, the host limit is fully supported and the field filter applies the limit.

If the host field is not explicitly included in the BY clause of the tstats search, the field filter limit is ignored, and the given field filter is applied across all the values of host fields, instead of any specific values of the host field.
source The field filter source limit is ignored and the field filter doesn’t apply the limit.

If the source field is explicitly included in the BY clause of the tstats search, the source limit is fully supported and the field filter applies the limit.

If the source field is not explicitly included in the BY clause of the tstats search, the field filter limit is ignored, and the given field filter is applied across all the values of source fields, instead of any specific values of the source field.
source type The field filter source type limit is ignored and the field filter doesn’t apply the limit. The field filter limit is ignored. All field filter limits are applied to all the values of source type in the entire index, not just the source type that is specified as the field filter limit.

For example, say you have a field filter on the _local index that redacts the ssn field, and sets a source limit for the sources called source1 and source2.

When you run the following search, the field filter will redact the value of the ssn field in events that originate from all sources in the index, not just source1 and source2.

CODE 
｜ tstats values(ssn) where index=user_ssn by user

However, if you run the following search, the field filter will only redact the value of the ssn field in events that originate from source1 and source2.

CODE 
｜ tstats values(ssn) where index=user_ssn by user, source

Field filters override fillnull_value in tstats searches

Field filters in tstats searches do not respect the fillnull_value argument, which replaces null values with a user-specified value for any field in the group-by list. When field filters are applied in tstats searches, they replace or delete sensitive fields and values depending on the field filter configuration, and ignore any fillnull_value settings in the search.

Field filters always delete fields and their values in certain tstats searches

Regardless of how the field filter replacement method is set, field filters always delete sensitive fields and their values when running searches with the tstats command in the following cases:

  • Field filters don't support field value replacement for searches that run the tstats command and the following aggregate functions on indexes protected by field filters:
    • avg(field)
    • min(field)
    • max(field)
    • range(field)
    • sum(field)
    • sumsq(field)
    • var(field)
    • varp(field)
    • stdev(field)
    • tdevp(field)

    Even if a field filter is set to hash or redact a field value, the field filter will instead delete the field and its value. Splunk software does this because these aggregate functions require numerical input, and these kinds of numeric functions should not be performed on sensitive fields protected by field filters. Deleting the field and its value ensures that functions on sensitive field data, such as sum(ssn) and sum(password), won’t appear in the aggregation output for these functions when used in tstats searches.

  • When searches run the tstats command with the prefix command in the format PREFIX(key=val) when the key represents a sensitive field. For example, in the following searches, the field filter deletes the result of PREFIX(ssn=12).
    CODE 
    | tstats count(PREFIX(ssn=12)) where index=main
    CODE 
    | tstats values(PREFIX(ssn=12)) where index=main

Field filter detection in tstats searches

Field filters in tstats searches detect sensitive fields in the where clause of your search after search filters have been applied. If a sensitive field appears anywhere in the final search, either because it is in your original tstats search or is introduced by a search filter , Splunk platform applies the corresponding field filter to fully protect the data.

Field filters apply their own limits on source, sourcetype, or host, independently of any scope specified in search filters. This design ensures comprehensive data protection because field filters are applied consistently whenever sensitive fields are present in the final search that runs, regardless of where those fields originate.

For example, say the following search filter is defined on the Restrictions tab for a new role called analyst:

NOT(source::/home/sensitive/data AND ssn::*)

In this example, the Splunk deployment also has a field filter called redact_ssn_with_string that replaces the value of the sensitive field ssn with the word redacted. The field filter is set on the tstats_fieldfilter index and is limited to the /home/app/log source.

Now let's say a user with the analyst role runs the following search:

CODE 
| tstats count where index=tstats_fieldfilter TERM(ssnhash=*)

The search filter adds the ssn field to the search, and the field filter is applied based on the final search that the Splunk platform runs:

CODE 
| tstats count where index=tstats_fieldfilter TERM(ssnhash=*)AND NOT(source=/home/sensitive/data AND ssn=*)

This search triggers the redact_ssn_with_string field filter because the final search contains the ssn field. Although the original search just referenced ssnhash and not ssn, the field filter is applied to the search according to its configured limits on the tstats_fieldfilter index and /home/app/log source. As a result, all ssnhash and ssn fields originating from those indexes and sources are protected when the analyst runs their search.