Create a ruleset with the Ingest Actions page

Data preview

Data preview is available when you're building a ruleset. Data preview can help you define rules. It also estimates the changes a rule will have on the data source.

The data preview is only a preview of the rule changes, and does not actually modify any indexed data.

You can use data preview with several types of data sources:

  • Live capture (deployment server and standalone indexers only) uses data directly from an incoming data stream.
  • Indexed data (not available for deployment server) uses recently indexed data.
  • Sample file uses data from a sample file that you upload. You can also copy and paste event logs.

Note: The Sourcetype field is case-sensitive. You must use the correct case to show results for the sample events.

Selecting Sample retrieves events from the indexers or the incoming data stream. The All Events tab provides a visual indication of the rule matches. The Affected Events tab provides a total count, and displays the full event for every rule match.

If your data uses renamed source types, you might encounter issues that require remediation. See the Splunk Lantern article Using ingest actions with source types that are renamed with props and transforms.

When using live capture with deployment server, ensure the following conditions are met:

  • Your firewall allows connections from the deployment server to its deployment clients.
  • The deployment server and deployment clients use the same pass4SymmKey in server.conf:
[deployment]
pass4SymmKey = <passphrase string>

Mask with regular expression

Use a masking rule to replace strings of text in your logs. A mask rule is typically applied to fields with unique identifiers, or user names, that are captured through logging.

The mask rule requires you to provide:

Setting Description
Match Regular Expression The field accepts a regular expression, or a simple string to match in the events.
Replace Expression The field accepts a string value to replace any matches. If you want to remove the matched values without substituting a replacement, simply enter a blank space.

Filter with regular expression

Use a filtering rule to remove entire events from your logs. A filter rule is typically applied to log events that are not valued, such as DEBUG messages, log headers, and redundant log messages.

This filter rule requires you to provide:

Setting Description
Source Field Use the drop down to select a data source by: _raw, host, index, source, or source type.
Drop Events Matching Regular Expression The field accepts a regular expression, or a simple string to match in the events.

When using a filter rule, the Affected Events tab is a preview of events that will be deleted once the ruleset is deployed. If you add another rule after a filter, the new rule applies to any remaining, unfiltered events only.

Filter with eval expression

Using an eval expression is an alternative to using a regular expression for filtering. In most cases, the eval syntax is easier to read and comprehend, while offering the same functionality as a regular expression.

Note: The eval expression rule does not support ingest-time lookups.

Use a filtering rule to remove entire events from your logs. A filter rule is typically applied to log events that are not valued, such as DEBUG messages, log headers, and redundant log messages.

This filter rule requires you to provide:

Setting Description
Drop Events Matching Eval Expression When the eval expression match is true, those events will be dropped.

When using a filter rule, the Affected Events tab is a preview of events that will be deleted once the ruleset is deployed. If you add another rule after a filter, the new rule applies to any remaining, unfiltered events only.

Set index

Use a set index rule to specify or change the destination index for an event routing to a Splunk destination. You can optionally filter the events that the rule applies to.

Note: If this rule does not apply to a particular Splunk destination event, that event goes to the index otherwise designated for the event, either the default "main" index or an index specified through the available layered configurations in the Splunk configuration system, for example, through settings in inputs.conf or outputs.conf.

You can either specify a string for the destination index name, or you can set the index based on an eval expression, which allows you to conditionally route to different indexes.

The set index rule includes these settings:

Setting Description
Condition Optionally filter the events that follow the set index rule.
Set index as Set the index to a string value (for example, "my_index") or use an eval expression to determine the index name based on specified conditions.

Route to Destination rule

Use a routing rule to select events, and split or duplicate them between one or more destinations.

This routing rule requires you to provide:

Setting Description
Condition Choose a method to match events for routing. Choose the regex or eval condition to select specific events, or none when you want all events sent to a destination. If a condition is set, only events matching the condition will be sent to the destination(s).
Immediately send to By default, the destination is "Default Destination". Any matching events are placed back into the Splunk Enterprise indexing queue for processing and indexing to a Splunk index, either on the local instance or on a downstream or associated instance, according to the deployment topology.

For example, in the case of a heavy forwarder, the default destination is an index on the indexer at the end of the chain of forwarders. Similarly, in the case of an indexer cluster, the default destination is an index on the peer nodes.

The destination index for each event is either the default index (main), the index determined by the configuration layers, if any, or an index determined by a set index rule.

The destination rule also supports AWS S3 and other S3-compliant destinations. You must configure an S3 remote storage destination before using the destination in a "Route to Destination" rule. See Create an S3 destination.

If more than one destination is chosen, a copy of any matching events is sent to all destinations chosen.

Clone events and apply more rules This toggle causes data ingest to create a clone of the event stream, applying the rules currently defined in the ruleset, and route the stream to the specified destination, while applying any additional newly defined rules against the event stream and routing that subset to a second specified destination, defined in a second Route to Destination rule. As with all rules, the ruleset must be saved and deployed before the destination rules start functioning.

Data Preview for Final Destination

The last rule in every ruleset sends any remaining events along the ingestion pipeline to the indexer for indexing. The rule offers an estimate of the data volume that will be indexed.

If you use the "Route to Destination" rule in your ruleset, this rule might be skipped. For example, if a Route to Destination rule includes "Immediately send to: Splunk Index," the data stream is split at the routing rule, and the matching events are sent to be indexed. In that scenario, the Final Destination rule will display a 0Kb indexed data estimate, despite events being sent for indexing from the routing rule.