Pipeline functions

Pipeline functions are SPL2 evaluation functions that are supported in Edge Processor and Ingest Processor pipelines only.

The following list contains SPL2 functions that are supported in Edge Processor and Ingest Processor pipelines only. These functions are not supported in searches.

batch_id()

The batch_id() function returns the ID associated with the current batch of events that the pipeline is processing.

Pipelines process events in batches. This function returns the ID associated with the current batch.

Usage

You can use this function with the eval and where commands, in the WHERE clause of the from command, and as part of evaluation expressions with other commands.

Examples

The following example creates an event field named processor_batch, which contains the ID of the batch that the event is associated with:

CODE 
... | eval processor_batch=batch_id()

You can then use the ID in other commands, such as stats. This next example uses the stats command to calculate the sum of the bytes_out field, and groups the sums by the batch ID stored in the processor_batch field:

CODE 
... | stats sum(bytes_out) by processor_batch

batch_time()

The batch_time() function returns the UNIX timestamp indicating the processing time of the current batch of events that the pipeline is processing.

Pipelines process events in batches. This function returns the UNIX timestamp, in seconds, indicating the processing time of the current batch.

Usage

You can use this function with the eval and where commands, in the WHERE clause of the from command, and as part of evaluation expressions with other commands.

Examples

The following example creates an event field named processing_time, which contains the timestamp indicating when the event was processed as part of a batch:

CODE 
... | eval processing_time=batch_time()

You can then use the timestamp in other commands, such as stats. This next example uses the stats command to calculate the sum of the bytes_out field, and groups the sums by the timestamp stored in the processing_time field:

CODE 
... | stats sum(bytes_out) by processing_time

instance_id()

The instance_id() function returns identifying information about the Edge Processor instance or Ingest Processor that processed the event.

This function returns identifying information about the Edge Processor instance or Ingest Processor that processed the event.

  • For Edge Processor instances, this function returns the host name of the machine.

  • For Ingest Processor, this function returns the UUID (universally unique identifier) associated with the Ingest Processor.

Usage

You can use this function with the eval and where commands, in the WHERE clause of the from command, and as part of evaluation expressions with other commands.

Examples

The following example creates an event field named processor_id, which contains the host name of the Edge Processor instance or the UUID of the Ingest Processor that processed the event:

CODE 
... | eval processor_id=instance_id()

After indexing these events, you can use the processor_id value to search for all events that were processed by a specific Edge Processor instance or Ingest Processor. For example, the following search returns all events in the main index that were processed by the Edge Processor instance hosted on the ip-10-210-34-60 machine:

CODE 
FROM main WHERE processor_id="ip-10-210-34-60"

As another example, the following search returns all events in the main index that were processed by the Ingest Processor associated with the UUID value 4b0c083a-97c0-4586-8fd7-356e08ab08d1:

CODE 
FROM main WHERE processor_id="4b0e083a-97e0-4586-8fd7-356e08ab08d1"