Edge Processor metrics reference

As an Edge Processor administrator, you can create dashboards to monitor the health, status, and information of the data that is flowing through your Edge Processors. You can do this from your Splunk platform deployment. For more information on creating dashboards, see About dashboards.

You can create these queries by using the Search & Reporting app in the Splunk platform. Aside from searching and creating dashboards to monitor your Edge Processor, you can configure alerts to notify specific changes that you would like to track. For more information on how to configure alerts in the Splunk platform, see Getting started with alerts.

S2S ACK performance benchmarks

The following results compare Splunk-to-Splunk (S2S) throughput with and without end-to-end acknowledgment (ACK) for standalone and hybrid Edge Processor deployments. Hybrid ACK results include the default one-connection test and a tuned test using 4–6 ingestion connections.

Table 1. S2S ACK benchmark results
Workload and metric Standalone, non-ACK Standalone, ACK Hybrid, non-ACK Hybrid, ACK (default, 1 connection) Hybrid, ACK (tuned, 4–6 connections)
Windows event log, events per second (EPS) 80,000 100,000 45,000 40,000 62,000
Windows event log, throughput (MB/s) 67.7 84.5 38 33.6 53
Windows event log, p95 ACK latency Not applicable 3–5 s (CCM) Not applicable 2.4–22 s (average 2.4–3.3 s) 2.3–17 s (average 1.6–2.5 s)
Syslog, events per second (EPS) 190,000 120,000 65,000 80,000 112,000
Syslog, throughput (MB/s) 16.8 10.4 5.7 6.3 9.8
Syslog, p95 ACK latency Not applicable About 10 s (CCM) Not applicable Not reported 3.7–5.7 s (average 1.1–1.5 s)
Note: These are benchmark results do not specify all test-environment details, so use the results as reference points, not capacity guarantees. ACK provides at-least-once delivery and can result in duplicates. Receiver-side ACK confirms that the batch reached the Edge Processor persistent queue; exporter-side ACK confirms indexer acknowledgment. Configure ACK on both legs for end-to-end ACK. See Configure your Edge Processor to use acknowledgment with Splunk forwarders for current configuration steps and prerequisites.

EP Hybrid performance benchmarks

EP Hybrid runs the Edge Processor alongside a Splunk heavy forwarder (HF). In this mode, Splunkd handles input and parsing before data passes to EP over the local IPC connection. The results below are from the EP 1.13.1 internal benchmark presentation and are workload-specific.

Table 2. S2S input benchmarks with default settings, without ACK
Workload and metric Standalone EP EP Hybrid Hybrid CPU and memory
Windows event log, EPS 80,000 (80 connections) 45,000 (1 connection) CPU about 330% total (EP 210%, HF 120%); memory about 4.1 GiB total (EP 2.7 GiB, HF 1.4 GiB)
Windows event log, throughput 67.7 MB/s 38 MB/s See resource use in the row above
Syslog, EPS 190,000 (190 connections) 65,000 (1 connection) CPU about 280% total (EP 190%, HF 90%); memory about 4.3 GiB total (EP 2.9 GiB, HF 1.4 GiB)
Syslog, throughput 16.8 MB/s 5.7 MB/s See resource use in the row above
Table 3. EP Hybrid non-ACK input path benchmarks
Input path Sample maximum EPS Throughput (MB/s) Conditions
TCP syslog to EP syslog 78,000 6.66 Untuned
TCP syslog to EP Windows event log 45,000 37.82 Untuned
UDP syslog to EP syslog 74,750 6.33 Untuned
UDP syslog to EP Windows event log 10,360 to about 41,000 8.77 to about 34.7 Untuned to kernel and buffer tuned
HEC to EP syslog 120,000 10.47 parallelIngestionPipelines=2
HEC to EP Windows event log 73,000 62 parallelIngestionPipelines=2
Table 4. EP Hybrid real-world S2S workload reference, without ACK
AWS instance type EP CPU / HF CPU EP memory / HF memory (GB) Throughput (MB/s) Ingest (TB/day) Ingestion pipelines / sent to EP (%)
c6a.large (2 vCPU, 4 GiB RAM) 130% / 50% 2.1 / 0.6 10.9 0.94 1 / 80.3%
c6a.xlarge (4 vCPU, 8 GiB RAM) 200% / 70% 2.3 / 0.6 22.5 1.94 2 / 82.9%
c6a.2xlarge (8 vCPU, 16 GiB RAM) 500% / 200% 2.6 / 0.8 38.1 3.29 4 / 87.7%

The real-world reference is a test using Palo Alto Networks firewall and URL logs sent from a heavy forwarder through a Splunk processing pipeline to EP. Its dataset and methodology differ from the other benchmark rows. Edge Processor sizing guidelines also describe internal S2S performance testing and advise evaluating throughput for the specific deployment.

Tuning results were conducted on an 8 vCPU, 16 GiB host, scaling ingestion pipelines for S2S with ACK increased Windows event log throughput from 33.6 to 53 MB/s (+58%) and syslog throughput from 6.3 to 9.8 MB/s (+55%). For HEC with two ingestion pipelines, syslog increased from 65,000 to 120,000 EPS (+85%) and Windows event log increased from 45,000 to 73,000 EPS (+62%). Increasing ingestion pipelines does not improve TCP or UDP syslog paths in this test because Splunk pins those paths to one pipeline. Setting maxKbps to 0 removes the default per-pipeline 256 kbps limit. UDP receive-buffer tuning increased Windows event log syslog from 10,360 to about 41,000 EPS, though packet-receive errors rose above about 20,000 EPS; the presentation recommends TCP syslog for events consistently larger than 1,400 bytes.

Note: Benchmark results depend on the event mix, pipeline configuration, connection count, HF and EP versions, host capacity, network, and destination. The figures are test observations, and not sizing guarantees. Hybrid mode uses HF resources in addition to EP resources. See Set up an Edge Processor in hybrid mode for deployment prerequisites and the current hybrid architecture.

Searching for Edge Processor metrics

The Edge Processor metrics are stored in the _metrics index. To access these metrics, use the mstats command. The following table contains the common query patterns used to search for these metrics.

Stats metric term When to use
Rate_sum()

Counters
Latest()

Gauges
Avg()

Averages over time
Perc95()

Latency percentiles

An example query syntax would look like the following:

mstats rate_sum(metric_name) WHERE index=_metrics span=1m

To see more information on Edge Processor example searches for common metrics, see Example searches for Edge Processor metrics.

Data processing metrics

Throughput metrics

Use these metrics to track how data moves from ingestion to destination.

Track the rate at which data is successfully processed through the Edge Processor instance over a specific period. The following table lists the metrics that track this information.

Metric Metric name Unit Label
Data received by processor processor_bytes_in_total Bytes dataset_name, evt_source_type, signal_type
Data sent by processor processor_bytes_out_total Bytes dataset_name, evt_source_type, signal_type
Events received processor_events_in_total Event count dataset_name, evt_source_type
Events sent processor_events_out_total Event count dataset_name, evt_source_type

Egress data metrics

Track the rate at which outbound data is sent from the Edge Processor to an external destination over a specific period. The following table lists the metrics that track this information.

Metric Metric name Unit Label
Bytes written to destination egress_bytes_written_total Bytes exporter_type, dataset_name, signal_type
Send attempts including retries write_to_sink_attempts_total Count exporter_type, dataset_name
Export errors export_destination_errors_total Count exporter_type, dataset_name, errorType
Write latency to destination write_to_sink_latency_seconds Histogram exporter_type, dataset_name, signal_type
Logs successfully sent exporter_sent_log_records Count exporter
Metrics successfully sent exporter_sent_metric_points Count exporter
Spans successfully sent exporter_sent_spans Count exporter
Failed log sends exporter_send_failed_log_records Count exporter
Failed metric sends exporter_send_failed_metric_points Count exporter
Failed span sends exporter_send_failed_spans Count exporter

Queue metrics

Track the data flowing through the queue and retries over a specific period. The following table lists the metrics that track this information.

Metric Metric name Unit Type
Current queue size (entries) queue_size_entries Count Gauge
Current queue size (bytes) queue_size_bytes Bytes Gauge
Maximum queue capacity queue_max_bytes Bytes Gauge
Undispatched entries queue_undispatched_entries Count Gauge
Bytes written into queue enqueued_entries Count Counter

System and host metrics

Process metrics

Use these metrics to track the health or performance of your host or machine.

Monitor the health of the process of your Edge Processor by using these metrics in your search. The following table lists the metrics that track this information.

Metric Metric name Unit Label
Process uptime process_uptime Seconds -
Physical memory usage process_memory_rss Bytes -
Heap memory allocated process_runtime_heap_alloc_bytes Bytes -
CPU time used process_cpu_seconds Seconds state

Host metrics

Monitor the health of your host by using these metrics in your search. The following table lists the metrics that track this information.

Metric Metric name Unit Label
CPU usage by state system.cpu.time, system.cpu.utilization Seconds, ratio CPU, state
RAM usage by state system.memory.usage, system.memory.utilization Bytes, ratio state
Disk I/O operations and throughput system.disk.io, system.disk.operations Bytes, count device, direction
File system capacity and usage system.filesystem.usage, system.filesystem.utilization Bytes, ratio device, mountpoint, type, state
Network traffic and errors system.network.io, system.network.errors, system.network.packets, Bytes, count device, direction

Metric label reference

Resource identification labels

The following tables contain corresponding labels for specific metrics.

Metric attribute Label Example
Unique supervisor instance ID service_instance_id sup-abc123
Supervisor group ID processor_id group-1
Tenant ID tenant tenant-xyz
Machine host name host_name edge-host-01

Pipeline and component labels

Metric attribute Label Example
Pipeline ID pipeline_id 7200faed6f099ed392c155e339e5ddea_1777916602202068260_rjlqv
Exporter component ID exporter S2S/shared.pipelines.default_splunk_cloud_destination
Type of exporter exporter_type awss3, splunkhec, splunk
Receiver component ID receiver “*hecreceiver”
Transport protocol transport http

Data classification labels

Metric attribute Label Example
Telemetry type signal_type logs, metrics
Dataset name dataset_name shared.pipelines.default_splunk_cloud_destination
Dataset kind dataset_kind hecreceiver
Source type evt_source_type access_combined
Agent hostname agent agent-host-01
Internal index flag internal True, False
Error category error_type Unclassified, DestinationError, ConnectionForbidden, HostNotFound, InvalidRequest. ResourceNotFound,

Example searches for Edge Processor metrics

The following table highlights the searches that you can use to create dashboards for Edge Processor metrics. For more information on how to create dashboards to monitor your Edge Processor health, see About dashboards.

Metrics Example search
Edge Processor data throughput overview | mstats sum(processor_bytes_in_total) as bytes_in sum(processor_bytes_out_total) as bytes_out sum(egress_bytes_written_total) as bytes_egress WHERE index=_metrics span=5m | eval in_mb = round(bytes_in / 1024 / 1024, 2) | eval out_mb = round(bytes_out / 1024 / 1024, 2) | eval egress_mb = round(bytes_egress / 1024 / 1024, 2) | timechart span=5m avg(in_mb) as "Processor Input (MB)" avg(out_mb) as "Processor Output (MB)" avg(egress_mb) as "Egress (MB)"
Edge Processor queue health | mstats latest(queue_size_bytes) as queue_bytes latest(queue_max_bytes) as queue_capacity WHERE index=_metrics by host_name span=1m | eval queue_usage=round(queue_bytes/queue_capacity*100, 2) | timechart max(queue_usage) as queue_usage by host_name useother=f usenull=f
Edge Processor processing latency | mstats rate(_value) as time_cost WHERE index=_metrics host="*" metric_name=*processor_processing_time_bucket by _timeseries, le, pipeline_id span=5m | stats sum(time_cost) as total_time_cost by _time, le, pipeline_id | sort _time, pipeline_id, le | eventstats max(total_time_cost) as total_hist_rate, last(le) as uppermost_bound, count as num_buckets by _time, pipeline_id | eval rank=exact(0.95)*total_hist_rate | streamstats current=f last(le) as gr, last(total_time_cost) as last_hist_rate by _time, pipeline_id | eval gr=if(isnull(gr), 0, gr), last_hist_rate=if(isnull(last_hist_rate), 0, last_hist_rate) | where total_time_cost >= rank | dedup _time, pipeline_id | eval res=case(lower(uppermost_bound) != "+inf" or num_buckets < 2, "NaN", lower(le) == "+inf", gr, gr == 0 and le <= 0, le, true(), exact(gr + (le-gr)*(rank - last_hist_rate) / (total_time_cost - last_hist_rate))) | fields _time, pipeline_id, res | rename res as "histperc" | timechart max(histperc) by pipeline_id
Edge Processor resource utilization | mstats latest(process_memory_rss) as memory latest(process_runtime_heap_alloc_bytes) as heap WHERE index=_metrics BY service_instance_id span=1m | eval memory_mb = round(memory / 1024 / 1024, 2) | eval heap_mb = round(heap / 1024 / 1024, 2) | timechart span=1m avg(memory_mb) as "RSS Memory (MB)" avg(heap_mb) as "Heap Memory (MB)"
Top datasets by volume in Edge Processor | mstats sum(processor_bytes_in_total) as bytes WHERE index=_metrics BY dataset_name span=1h | eval gb = round(bytes / 1024 / 1024 / 1024, 3) | stats sum(gb) as total_gb by dataset_name | sort -total_gb | head 10
Edge Processor exporter comparison | mstats sum(egress_bytes_written_total) as bytes perc95(write_to_sink_latency_seconds) as latency sum(exporter_errors_total) as errors WHERE index=_metrics BY exporter_type span=5m | eval gb = round(bytes / 1024 / 1024 / 1024, 3) | stats avg(gb) as avg_gb avg(latency) as avg_latency_sec sum(errors) as total_errors by exporter_type
Edge Processor agent ingestion breakdown | mstats sum(processor_bytes_in_by_agent_total) as bytes WHERE index=_metrics BY agent span=1h | eval gb = round(bytes / 1024 / 1024 / 1024, 3) | stats sum(gb) as total_gb by agent | sort -total_gb