dedup command: Examples
The following are examples for using the SPL2 dedup command.
To learn more about the SPL2 dedup command, see How the SPL2 dedup command works.
1. Remove duplicate results based on one field
Remove duplicate search results with the same host value.
... | dedup host2. Keep the first 3 duplicate results
For search results that have the same source value, keep the first 3 that occur and remove all subsequent results.
... | dedup 3 source3. Sort events in ascending order before removing duplicate values
Use the order by clause in the from command to sort the events by time in ascending order, the default order. Sorting the events ensures that the oldest events are listed first. Remove duplicate results with the same source value. Only the oldest events are retained.
| from main order by ASC _time | dedup source4. Sort events after removing duplicate values
Remove duplicate search results with the same host value and sort the events by the _size field in descending order.
... | dedup host | sort -_size5. Keep results that have the same combination of values in multiple fields
For search results that have the same combination of source AND host values, keep the first 2 that occur and remove all subsequent results.
... | dedup 2 source, host6. Remove only consecutive duplicate events
Remove only consecutive duplicate events. Keep non-consecutive duplicate events. In this example duplicates must have the same combination of values the source and host fields.
... | dedup consecutive=true source, hostSee also
dedup command