Aggregate event data using Ingest Processor

Create a pipeline to aggregate the data being processed by the Ingest Processor.

1. Navigate to the Pipelines page and then select New pipeline, then Ingest Processor pipeline.
2. Follow the on-screen instructions to define a partition, optionally enter sample data, and select a data destination. For detailed instructions, see Create pipelines for Ingest Processor.

   Be aware of the following when creating a pipeline for aggregations:

   • Your sample data must contain accurate examples of the values that you want to summarize into aggregations. For example, the following sample events represent requested files from a server and contains aggregable data:

     server_name	server_ip	file_requested	bytes_out	sourcetype
     web-01	10.1.2.3	/index.html	500	web
     web-01	10.1.2.3	/images/logo.png	8500	web
     web-02	10.1.2.4	/css/main.css	1200	web

   • On the Select a destination page, if you select a Splunk platform destination, you can configure index routing. As a best practice, avoid sending both non-aggregated and aggregated data to the same index.

   After you complete the on-screen instructions, the pipeline editor displays the SPL2 statement for your pipeline.
3. Select the plus icon () in the Actions section and then select Summarize data.
   The Summarize data dialog box opens. By default, the Aggregations section shows an unconfigured aggregation.
4. (Optional) Set the Maximum delay option to the maximum length of time for which the Ingest Processor can hold and aggregate incoming events before it emits the result.
5. (Optional) If you plan to group your aggregations by the host, source, sourcetype, or index metadata fields, then set Output mode to one of the following:

   Output mode name	Description
   Summary mode	The Ingest Processor adds the prefix orig_ to the names of those metadata fields, so that you can use them to differentiate aggregated data from non-aggregated data.
   Passthrough mode	The Ingest Processor does not change the names of those metadata fields.

6. In the Aggregations section, do the following:
   1. Set the Field option to the name of the event field that you want to aggregate.
   2. Set the Calculation option to the type of calculation that you want to perform on the specified field.
   3. (Optional) To specify the name of the event field that stores the aggregated data, select the Edit alias () icon and then enter the desired field name.
   4. (Optional) You can configure multiple aggregations by selecting the Add aggregations () icon and then repeating steps 7a-c.
   For example, using the sample data shown in step 2, you can calculate the sum of the values in the bytes_out field by setting the Field option to bytes_out and the Calculation option to Sum. To keep the name of the aggregated field as bytes_out instead of changing it to sum(bytes_out), set the alias of the aggregation to bytes_out.
7. (Optional) In the Group by section, you can specify one or more fields to group the aggregations by. Do the following:
   1. Select the Add group by () icon.
   2. In the new entry that appears in the Group by section, set the Field option to the name of the event field that you want to group the aggregations by.
   3. (Optional) Repeat steps 8a-b as needed to define more groupings for the aggregations.
   Continuing the example from step 7, you can group the sum of bytes_out by the server names given in the server_name field. To do this, set the Field option to server_name.
8. Select Apply.

   The pipeline editor adds a stats command to your pipeline. If you added sample data to your pipeline, the preview results panel shows the results of the aggregation.

   For example, if you used the sample data and example configurations described in the preceding steps, the pipeline editor shows the following SPL2 statement:

   PYTHONCopy

   $pipeline = | from $source | @maxdelay("30seconds") stats mode=summary sum(bytes_out) as bytes_out by server_name | into $destination;

   $pipeline = | from $source
   | @maxdelay("30seconds") stats mode=summary sum(bytes_out) as bytes_out by server_name
   | into $destination;

   The aggregated results look like this:

   server_name	bytes_out	_raw
   web-01	9000	{"server_name": "web-01", "bytes_out": 9000}
   web-02	1200	{"server_name", "web-02", "bytes_out": 1200}

   Note: Your pipeline keeps the event fields specified in the Aggregations and Group by sections and drops the unspecified fields. In the example above, the pipeline drops the server_ip and file_requested fields because they are not included in the aggregation.
9. (Optional) If you plan to work with the aggregated data in the Splunk platform using operations that require the data to be in prestats format, then add the prestats argument to the stats command. In the SPL2 editor, do one of the following:
   • To store prestats data values in the _raw field, type prestats=raw inside the stats command, in the space before the aggregation expression. For example:
     PYTHONCopy

     $pipeline = | from $source | stats prestats=raw mode=summary sum(bytes_out) as bytes_out by server_name | into $destination;

     $pipeline = | from $source
     | stats prestats=raw mode=summary sum(bytes_out) as bytes_out by server_name 
     | into $destination;

   • To store prestats data values in top-level event fields, type prestats=fields inside the stats command, in the space before the aggregation expression. For example:
     PYTHONCopy

     $pipeline = | from $source | stats prestats=fields mode=summary sum(bytes_out) as bytes_out by server_name | into $destination;

     $pipeline = | from $source
     | stats prestats=fields mode=summary sum(bytes_out) as bytes_out by server_name 
     | into $destination;

10. Save your pipeline, and then apply it to your Ingest Processor. For more information, see Apply a pipeline.