SPL2 pipeline templates reference
The Edge Processor and Ingest Processor solutions come with a selection of prebuilt SPL2 pipeline templates to help you manipulate, route, and analyze your data. You can edit these templates for your specific use case. See the following list:
| Data source | Pipeline template name | Description | Edge Processor | Ingest Processor |
|---|---|---|---|---|
| Amazon Virtual Private Cloud (VPC) | AWS VPC Flow logs: Filter unwanted logs and generate metrics from logs | Generate metrics with dimensions and filter some events from VPC flow logs. | Yes | Yes |
| AWS CloudTrail | AWS CloudTrail Logs: Reduce log size | Reduce the size of AWS CloudTrail events while preserving compatibility with the Splunk Common Information Model (CIM) and security detections. | Yes | Yes |
| Azure Monitor | Azure Monitor AAD Logs: Reduce log size | Reduce the size of Azure Monitor AAD logs emitted by the Splunk Add-on for Microsoft Cloud Services by removing unnecessary data. The compatibility with the Splunk Common Information Model (CIM) and Security Detections is preserved. | Yes | Yes |
| Cisco ASA | Cisco ASA : Filter out noise from Cisco ASA log events using `noisereduce` function | Automate noise reduction for Cisco ASA logs by leveraging Splunk knowledge objects, improving downstream analytics and alerting. | Yes | Yes |
| Cisco ASA | Cisco ASA log reduction: Remove some fields, drop unnecessary events | Reduce the size of Cisco ASA logs while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Cisco ASA | Cisco ASA syslog data: Extract and filter cisco asa syslog data | Take Cisco ASA syslog message data and filter it. This template also automatically removes the header information from messages, which reduces the message size by 10%. This template will not filter messages with a syslog message ID of 430003. | Yes | Yes |
| CrowdStrike Falcon Data Replicator (FDR) | CrowdStrike FDR log reduction | Reduce the size of CrowdStrike FDR sensor event logs while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Generic template to get started | Drop fields: Drops the unnecessary fields from tabular formatted data | Drop specific values from tabular formatted data. This template can drop values that are separated by any delimiter, including commas, tabs, and pipes. | Yes | Yes |
| Generic template to get started | Empty field removal: Removes empty, null or blank fields from a JSON object | Remove empty, null, or blank fields from the top-level fields of a JSON object. | No | Yes |
| Generic template to get started | Field reduction and format conversion: Retains only mentioned fields and changes the format of the events | Convert key-value pairs into delimiter-separated events, and drop the unwanted fields from the events. | No | Yes |
| Generic template to get started | Generic data: De-identify Personally Identifiable Information | This template de-identifies Personally Identifiable Information (PII) from patient data. | Yes | Yes |
| Generic template to get started | Generic data: Mask IP addresses from a specific range | This template masks IP addresses based on a specified CIDR range. | Yes | Yes |
| Generic template to get started | Generic data: Route 'root' user events to special index | This template routes events related to the "root" user to a special index. | Yes | Yes |
| Generic template to get started | KV reduction: Normalizes the key-value fields from the provided json | Take JSON arrays of objects where each object stores keys and values as separate properties, and reduce those arrays into JSON objects that store each key-value pair as a property. | Yes | Yes |
| Generic template to get started | Lookup Field Replacement: Enrich events with CIDR metadata | Align an IP address to a CIDR range in order to enrich events from a lookup. | Yes | Yes |
| Generic template to get started | Lookup Field Replacement: Replace any repetitive data using a lookup to reduce the size of data | Reduce the size of CSV data by using a lookup to replace any repetitive data. | Yes | Yes |
| Generic template to get started | Multivalue Expansion: Expand multivalue fields and correlate with top level fields | Expand an array of metric values that represent unique entities, keeping the top-level fields intact and correlated to each metric value. | Yes | Yes |
| Generic template to get started | Retain specific fields: Retains only mentioned fields from the given JSON object | Choose from 3 different approaches to retain specific fields from JSON objects. | Yes | Yes |
| Generic template to get started | Sampling the data: Only keeps certain amount of events | Select a sample of the events that are passing through the pipeline by using random number generation. | Yes | Yes |
| Generic template to get started | Size calculation: Calculates the size for any fields | Measure the memory footprint of an event or individual field by capturing and previewing changes to event size as the events pass through the pipeline. Note: This template is designed to help you develop and test your pipelines. Use this template for performance tuning or resource optimization. | Yes | Yes |
| Generic template to get started | Time calculation: Calculates the processing time for pipeline of individual function | Calculate the processing time of the whole pipeline or an individual function. Note: This template is designed to help you develop and test your pipelines. Use this template to measure execution time in order to trace bottlenecks, monitor performance, and guide optimization efforts. | Yes | Yes |
| JSON | JSON data: Generate metrics from log data | Take pre-configured JSON data to show how the logs_to_metrics function can be used to convert logs to metrics. | No | Yes |
| Microsoft Office 365 (O365) | O365 Management Activity: Reduce log size (CIM & ESCU compatibility) | Reduce noise in O365 Management Activity events while preserving full compatibility with both the Splunk Common Information Model (CIM) and Splunk Enterprise Security Content Updates (ESCU). | Yes | Yes |
| Microsoft Office 365 (O365) | O365 Management Activity: Reduce log size (CIM only compatibility) | Reduce noise in O365 Management Activity events while preserving compatibility with only the Splunk Common Information Model (CIM). This template removes fields that are specific to Splunk Enterprise Security Content Updates (ESCU). | Yes | Yes |
| Okta Identity Cloud | Okta Logs: Reduce log size | Reduce the size of Okta logs while preserving compatibility with the Splunk Common Information Model (CIM) and security detections. | Yes | Yes |
| Palo Alto | Palo Alto Network logs: Reduce log size | Reduce the size of Palo Alto Network logs by removing unnecessary fields. Then, extract recommended event fields. | Yes | Yes |
| Palo Alto | Palo Alto Networks PAN-OS syslog data: Extract fields and classification of Palo Alto logs | Take Palo Alto Networks syslog message data and set the sourcetypes and indexes based on the message text. This pipeline also automatically removes the header information from messages, which reduces the message size by 10%. | Yes | Yes |
| Palo Alto | Palo Alto Network traffic logs: Generate metrics from logs | Generate metrics with dimensions from Palo Alto Network traffic logs, and then route the metrics and the original logs to two different destinations. | No | Yes |
| Kubernetes | Prometheus-formatted Kubernetes logs: Extract fields and generate metrics | Generate metrics with dimensions from Prometheus-formatted Kubernetes logs, and then route the metrics and the original logs to two different destinations. | No | Yes |
| Syslog | Syslog data: Extract fields and filter for systemd logs | Take syslog data and filter it for systemd events. | Yes | Yes |
| Syslog | Syslog data: Mask IP addresses from hostname field | Take syslog data and mask IP addresses from the hostname field. | Yes | Yes |
| Nix | UNIX and Linux bandwidth logs: Reduce log size and convert to TSV format | Reduce the size of 'bandwidth' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux cpu logs: Reduce log size and convert to TSV format | Reduce the size of 'cpu' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux df logs: Reduce log size and convert to TSV format | Reduce the size of 'df' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. The original tab-separated values (TSV) format of the logs and compatibility with the Splunk Common Information Model (CIM) are both preserved. | Yes | Yes |
| Nix | UNIX and Linux hardware logs: Reduce log size and convert to tab-separated key-value pair format | Reduce the size of 'hardware' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated key-value pair format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux interfaces logs: Reduce log size and convert to TSV format | Reduce the size of 'interfaces' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux iostat logs: Reduce log size and convert to TSV format | Reduce the size of 'iostat' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux lastlog logs: Reduce log size and convert to TSV format | Reduce the size of 'lastlog' logs emitted by the Splunk Add-on for Unix and Linux by converting the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux lsof logs: Reduce log size and convert to TSV format | Reduce the size of 'lsof' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux netstat logs: Reduce log size and convert to TSV format | Reduce the size of 'netstat' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux package logs: Reduce log size and convert to TSV format | Reduce the size of 'package' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux ps logs: Reduce log size and convert to TSV format | Reduce the size of 'ps' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux top logs: Reduce log size and convert to TSV format | Reduce the size of 'top' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated values (TSV) format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux vmstat logs: Reduce log size and convert to tab-separated key-value pair format | Reduce the size of 'vmstat' logs emitted by the Splunk Add-on for Unix and Linux by removing unnecessary data. Then, convert the logs into tab-separated key-value pair format while preserving compatibility with the Splunk Common Information Model (CIM). | Yes | Yes |
| Nix | UNIX and Linux process status logs: Generate metrics from logs | Generate metrics with dimensions from UNIX and Linux process logs, and then route the metrics and original logs to two different destinations | No | Yes |
| Windows | Windows event logs: Convert logs from XML to JSON | Convert Windows event logs from XML to JSON, reduce the size of the logs by removing unnecessary data, and extract event fields to ensure compatibility with the Splunk Add-on for Microsoft Windows and the Splunk Common Information Model (CIM). | Yes | Yes |
| Windows | Windows PerfMon Logs: Generate metrics from logs | Generate metrics with dimensions from Windows Perfmon logs, and then route the metrics to the destination. | Yes | Yes |