Performance

Database Monitoring relies on data (metrics and logs) that you send to Splunk Observability Cloud through an OpenTelemetry Collector (“collector”). You set up the collector to monitor your database platforms, and you use a different configuration for the collector for each different database platform type that you monitor. This topic explains:

  • Performance testing results: The performance impact of the collector’s actions, in other words the data collection overhead, on the database servers it monitors, and how we measured this impact.
  • Best practices: Our suggestions for optimal configuration for the collector. An optimal configuration is one that minimizes the impact on both the monitored database servers and on the collector’s host machine.
Microsoft SQL Server

Test setup

SettingValue
Collector host machineAWS m5.large container (used for running collector monitoring multiple databases)
Database host machineAWS RDS instance, db.m5.large (MS SQL Server 2016 Standard Edition)
Database typeMS SQL Server 2016 Standard Edition
Collector versionSplunk 0.131.1
Collection interval10 seconds
TopX count200
TopX collection interval60 seconds
Sample size1000
Lookback20 seconds
Load generation toolHammerDB (TPC-C workload)
Number of target databasesUp to 30 databases monitored concurrently by one collector instance
Collector memory allocationMinimum 2048 MiB to avoid forced garbage collection
VM tested5a.large

Performance testing results

  • The collector's CPU impact on the db.m5.large database host machine is less than 1% on idle systems. On systems under heavy load, the collector’s impact remains small and generally indistinguishable from normal CPU usage fluctuations.
  • The collector can monitor up to about 30 databases per instance on a properly sized VM with at least 2048 MiB memory to avoid performance issues.
  • CPU usage on the collector host remains low (around 5.21% average) even when monitoring 30 databases, allowing multiple collector instances (up to 7 or 8) to run on a single VM depending on CPU capacity.
  • The collector runs queries on the database, but the overhead from these queries is low enough to be effectively invisible in typical operational environments.

Best practices

  • Use a single collector instance to monitor a maximum of 30 database servers concurrently. This limit also reduces the single collector configuration file to a maximum of 30 sets of parameters, which helps you to manage its complexity.
  • Run each collector on a host machine with memory and CPU that meets or exceeds an AWS m5.large instance (2 vCPUs, 8GBB RAM) to avoid forced garbage collection and data loss. Proper memory allocation (at least 2048 MiB) is critical to avoid performance issues on the collector host.
  • Based on the observation that a single collector can monitor up to about 30 databases per instance, it is plausable that you can run multiple instances of a collector on a single host machine depending on the machine's CPU capacity.
  • Recommended collection settings for the collector:
    • 10-second collection interval
    • TopX count of 200
    • TopX collection interval of 60 seconds
    • Sample size of 1000
    • Lookback window of 20 seconds
Oracle Database

Test setup

SettingValue
Collector host machineAWS m5.large container (used for running collector monitoring multiple databases)
Database host machineAWS RDS Oracle instance, db.m5.large (Oracle 19c Enterprise Edition)
Database typeOracle 19c Enterprise Edition
Collector versionSplunk 0.131.1
Collection interval10 seconds
TopX count200
TopX collection interval60 seconds
Sample size1000
Lookback20 seconds
Load generation toolHammerDB (TPC-C workload)
Number of target databasesUp to 30 databases monitored concurrently by one collector instance
Collector memory allocationMinimum 512 MiB to avoid forced garbage collection

Performance testing results

  • The collector's CPU impact on the db.m5.large database host machine is less than 0.63% on idle systems. On systems under heavy load, the collector’s impact remains small and generally indistinguishable from normal CPU usage fluctuations.
  • The collector can monitor up to about 30 databases per instance on a properly sized VM with at least 512 MiB memory to avoid performance issues.
  • CPU usage on the collector host remains low (around 5.25% average) even when monitoring 30 databases, allowing multiple collector instances (up to 7 or 8) to run on a single VM depending on CPU capacity.
  • The collector runs queries on the database, but the overhead from these queries is low enough to be effectively invisible in typical operational environments.

Best practices

  • Use a single collector instance to monitor a maximum of 30 database servers concurrently. This limit also reduces the single collector configuration file to a maximum of 30 sets of parameters, which helps you to manage its complexity.
  • Run each collector on a host machine with memory and CPU that meets or exceeds an AWS m5.large instance (2 vCPUs, 8GB RAM) to avoid forced garbage collection and data loss. Proper memory allocation (at least 512 MiB) is critical to avoid performance issues on the collector host.
  • Based on the observation that a single collector can monitor up to about 30 databases per instance, it is plausable that you can run multiple instances of a collector on a single host machine depending on the machine's CPU capacity.
  • Recommended collection settings for the collector:
    • 10-second collection interval
    • TopX count of 200
    • TopX collection interval of 60 seconds
    • Sample size of 1000
    • Lookback window of 20 seconds