Edge Processor metrics reference
As an Edge Processor administrator, you can create dashboards to monitor the health, status, and information of the data that is flowing through your Edge Processors. You can do this from your Splunk platform deployment. For more information on creating dashboards, see About dashboards.
You can create these queries by using the Search & Reporting app in the Splunk platform. Aside from searching and creating dashboards to monitor your Edge Processor, you can configure alerts to notify specific changes that you would like to track. For more information on how to configure alerts in the Splunk platform, see Getting started with alerts.
Searching for Edge Processor metrics
The Edge Processor metrics are stored in the _metrics index. To access these metrics, use the mstats command. The following table contains the common query patterns used to search for these metrics.
| Stats metric term | When to use |
|---|---|
Rate_sum() |
Counters |
Latest() |
Gauges |
Avg() |
Averages over time |
Perc95() |
Latency percentiles |
An example query syntax would look like the following:
mstats rate_sum(metric_name) WHERE index=_metrics span=1m
To see more information on Edge Processor example searches for common metrics, see Example searches for Edge Processor metrics.
Data processing metrics
Throughput metrics
Use these metrics to track how data moves from ingestion to destination.
Track the rate at which data is successfully processed through the Edge Processor instance over a specific period. The following table lists the metrics that track this information.
| Metric | Metric name | Unit | Label |
|---|---|---|---|
| Data received by processor | processor_bytes_in_total |
Bytes | dataset_name, evt_source_type, signal_type |
| Data sent by processor | processor_bytes_out_total |
Bytes | dataset_name, evt_source_type, signal_type |
| Events received | processor_events_in_total |
Event count | dataset_name, evt_source_type |
| Events sent | processor_events_out_total |
Event count | dataset_name, evt_source_type |
Egress data metrics
Track the rate at which outbound data is sent from the Edge Processor to an external destination over a specific period. The following table lists the metrics that track this information.
| Metric | Metric name | Unit | Label |
|---|---|---|---|
| Bytes written to destination | egress_bytes_written_total |
Bytes | exporter_type, dataset_name, signal_type |
| Send attempts including retries | write_to_sink_attempts_total |
Count | exporter_type, dataset_name |
| Export errors | export_destination_errors_total |
Count | exporter_type, dataset_name, errorType |
| Write latency to destination | write_to_sink_latency_seconds |
Histogram | exporter_type, dataset_name, signal_type |
| Logs successfully sent | exporter_sent_log_records |
Count | exporter |
| Metrics successfully sent | exporter_sent_metric_points |
Count | exporter |
| Spans successfully sent | exporter_sent_spans |
Count | exporter |
| Failed log sends | exporter_send_failed_log_records |
Count | exporter |
| Failed metric sends | exporter_send_failed_metric_points |
Count | exporter |
| Failed span sends | exporter_send_failed_spans |
Count | exporter |
Queue metrics
Track the data flowing through the queue and retries over a specific period. The following table lists the metrics that track this information.
| Metric | Metric name | Unit | Type |
|---|---|---|---|
| Current queue size (entries) | queue_size_entries |
Count | Gauge |
| Current queue size (bytes) | queue_size_bytes |
Bytes | Gauge |
| Maximum queue capacity | queue_max_bytes |
Bytes | Gauge |
| Undispatched entries | queue_undispatched_entries |
Count | Gauge |
| Bytes written into queue | enqueued_entries |
Count | Counter |
System and host metrics
Process metrics
Use these metrics to track the health or performance of your host or machine.
Monitor the health of the process of your Edge Processor by using these metrics in your search. The following table lists the metrics that track this information.
| Metric | Metric name | Unit | Label |
|---|---|---|---|
| Process uptime | process_uptime |
Seconds | - |
| Physical memory usage | process_memory_rss |
Bytes | - |
| Heap memory allocated | process_runtime_heap_alloc_bytes |
Bytes | - |
| CPU time used | process_cpu_seconds |
Seconds | state |
Host metrics
Monitor the health of your host by using these metrics in your search. The following table lists the metrics that track this information.
| Metric | Metric name | Unit | Label |
|---|---|---|---|
| CPU usage by state | system.cpu.time, system.cpu.utilization |
Seconds, ratio | CPU, state |
| RAM usage by state | system.memory.usage, system.memory.utilization |
Bytes, ratio | state |
| Disk I/O operations and throughput | system.disk.io, system.disk.operations |
Bytes, count | device, direction |
| File system capacity and usage | system.filesystem.usage, system.filesystem.utilization |
Bytes, ratio | device, mountpoint, type, state |
| Network traffic and errors | system.network.io, system.network.errors, system.network.packets, |
Bytes, count | device, direction |
Metric label reference
Resource identification labels
The following tables contain corresponding labels for specific metrics.
| Metric attribute | Label | Example |
|---|---|---|
| Unique supervisor instance ID | service_instance_id |
sup-abc123 |
| Supervisor group ID | processor_id |
group-1 |
| Tenant ID | tenant |
tenant-xyz |
| Machine host name | host_name |
edge-host-01 |
Pipeline and component labels
| Metric attribute | Label | Example |
|---|---|---|
| Pipeline ID | pipeline_id |
7200faed6f099ed392c155e339e5ddea_1777916602202068260_rjlqv |
| Exporter component ID | exporter |
S2S/shared.pipelines.default_splunk_cloud_destination |
| Type of exporter | exporter_type |
awss3, splunkhec, splunk |
| Receiver component ID | receiver |
“*hecreceiver” |
| Transport protocol | transport |
http |
Data classification labels
| Metric attribute | Label | Example |
|---|---|---|
| Telemetry type | signal_type |
logs, metrics |
| Dataset name | dataset_name |
shared.pipelines.default_splunk_cloud_destination |
| Dataset kind | dataset_kind |
hecreceiver |
| Source type | evt_source_type |
access_combined |
| Agent hostname | agent |
agent-host-01 |
| Internal index flag | internal |
True, False |
| Error category | error_type |
Unclassified, DestinationError, ConnectionForbidden, HostNotFound, InvalidRequest. ResourceNotFound, |
Example searches for Edge Processor metrics
Resource identification labels
The following table highlights the searches that you can use to create dashboards for Edge Processor metrics. For more information on how to create dashboards to monitor your Edge Processor health, see About dashboards.
| Metrics | Example search |
|---|---|
| Edge Processor data throughput overview | | mstats sum(processor_bytes_in_total) as bytes_in sum(processor_bytes_out_total) as bytes_out sum(egress_bytes_written_total) as bytes_egress WHERE index=_metrics span=5m | eval in_mb = round(bytes_in / 1024 / 1024, 2) | eval out_mb = round(bytes_out / 1024 / 1024, 2) | eval egress_mb = round(bytes_egress / 1024 / 1024, 2) | timechart span=5m avg(in_mb) as "Processor Input (MB)" avg(out_mb) as "Processor Output (MB)" avg(egress_mb) as "Egress (MB)" |
| Edge Processor queue health | | mstats latest(exporter_queue_size) as queue_size latest(exporter_queue_capacity) as queue_capacity WHERE index=_metrics BY exporter span=1m | eval utilization_pct = round((queue_size / queue_capacity) * 100, 2) | timechart span=1m avg(utilization_pct) by exporter |
| Edge Processor processing latency | | mstats latest(exporter_queue_size) as queue_size latest(exporter_queue_capacity) as queue_capacity WHERE index=_metrics BY exporter span=1m | eval utilization_pct = round((queue_size / queue_capacity) * 100, 2) | timechart span=1m avg(utilization_pct) by exporter |
| Edge Processor resource utilization | | mstats latest(process_memory_rss) as memory latest(process_runtime_heap_alloc_bytes) as heap WHERE index=_metrics BY service_instance_id span=1m | eval memory_mb = round(memory / 1024 / 1024, 2) | eval heap_mb = round(heap / 1024 / 1024, 2) | timechart span=1m avg(memory_mb) as "RSS Memory (MB)" avg(heap_mb) as "Heap Memory (MB)" |
| Top datasets by volume in Edge Processor | | mstats sum(processor_bytes_in_total) as bytes WHERE index=_metrics BY dataset_name span=1h | eval gb = round(bytes / 1024 / 1024 / 1024, 3) | stats sum(gb) as total_gb by dataset_name | sort -total_gb | head 10 |
| Edge Processor exporter comparison | | mstats sum(egress_bytes_written_total) as bytes perc95(write_to_sink_latency_seconds) as latency sum(exporter_errors_total) as errors WHERE index=_metrics BY exporter_type span=5m | eval gb = round(bytes / 1024 / 1024 / 1024, 3) | stats avg(gb) as avg_gb avg(latency) as avg_latency_sec sum(errors) as total_errors by exporter_type |
| Edge Processor agent ingestion breakdown | | mstats sum(processor_bytes_in_by_agent_total) as bytes WHERE index=_metrics BY agent span=1h | eval gb = round(bytes / 1024 / 1024 / 1024, 3) | stats sum(gb) as total_gb by agent | sort -total_gb |