Metrics indexing performance
This topic summarizes the results of metrics indexing performance.
Size on disk
When ingesting typical metrics payloads with supported metrics source types (collectd_http
, statsd
, metrics_csv
), a metrics index requires about 50% less disk storage space compared to storing the same payload in an events index.
Throughput
Consider the following when deciding whether to scale horizontally by adding additional indexers.
Using the collectd_http
source type with an HTTP Event Collector (HEC) input, testing reached a constant of around 55,000 events per second maximum ingestion throughput, and around 58,000 events per second without additional search load.
- The default batch size was 5,000 events per batch. A significant difference in ingestion performance was not observed between batch sizes of 100 to 5,000 events.
- The
keep-alive
setting was enabled for these tests. - A typical event size was about 214 bytes.
Using the statsd
source type with a UDP input, throughput was highly variable depending on other network activity. For UDP inputs we recommend using a universal forwarder as close as possible to where metrics are collected.
Speed
Consider the results from the following test for running metrics queries. This test used metrics from 1,000 hosts, with a total event count of 6 billion events in the metrics index, where queries were representative and did not use wildcards in metric_name
.
Time range | Events | Query speed |
---|---|---|
1 hour | 35 million | < 0.1s |
1 day | 850 million | ~3-5s |
1 week | 6 billion | ~20-22s |
See the Capacity Planning Manual.