If Splunk DB Connect retrieves a large amount of data from your database, it might affect your database performance, especially for the initial run. Subsequent runs of the same query might have less impact, as the database might cache results and only retrieve new data since the previous run of the query.

To use Splunk DB Connect in a distributed search environment, including search head clusters, you must determine the planned use cases. For ad hoc, interactive usage of database connections by live users, install the app on search heads. For scheduled indexing from databases and output of data to databases, install the app on heavy forwarders.

• Total number of Forwarders in the deployment, and the hardware specifications of each.
• Total expected data volume to transfer.
• Number of database inputs per Forwarder.
• Dataset size, per input, per interval.
• Execution Frequency, the interval length between a database input's separate executions.
• Fetch size (Not all JDBC drivers use this parameter for returning result sets).

Overloading the system can lead to data loss, so performance measurement and tuning can be critical. Use performance expections as the reference to plan your deployment, and monitor expected data returns for loss conditions.

This section provides measured throughput data achieved under certain operating conditions. Use the information here as a basis for estimating and optimizing the DB Connect throughput performance in your own production environment. As performance might vary based on user characteristics, application usage, server configurations, and other factors, Splunk can't guarantee specific performance results.

• Server: 8-core 2.60GHz CPU with hyper-threading enabled, providing 16 virtual CPUs (vCPUs), 16GB RAM, 1Gb Ethernet NIC, 64bit Linux
• JVM config: MaxHeapSize = 4GB. (For more information about the JVM memory setting, see "Performance tuning advice".)
• Data Source: Oracle 11g

During testing, varying the following factors had a negligible effect on performance:

• There was no discernable performance difference between running in batch mode (all events processed) and running in rising column mode (only the new events processed) with the same dataset.
• The number of defined database connections does not limit performance. The number of connections is different from the number of database inputs.

Since version 3.10.0 we use Splunk Modular Inputs, so the input execution is triggered by calling a REST API, then it is processed using Java concurrent features.

Depending on the instance resources (CPU and memory) where DB Connect is running, users may observe performance degradation when adding more inputs.This signals the need to consider scaling. For our simplest architectures: standalone Splunk or Heavy Forwarder, we offer the following ways of scaling:

1. Tuning DB Connect configuration - adjusting the configuration to handle higher workloads as described in section below.
2. Vertical scaling - add more resources to the instance running DB Connect or migrating to an instance with more resources.
3. Horizontal scaling - adding additional instances with DB Connect and splitting inputs between available instances.

Please Note: Increasing the number of CPUs will have positive impact on the number of threads which can concurrently execute inputs, but this also has some limits so observing resource usage is also crucial to find out if we are getting desired DB Connect performance vs instance resources usage

Results from our lab have shown that, for instance, with 16vCPUs, 64GiB RAM it was safe to process approximately ~1000 inputs. The best performance was achieved with a medium workload configuration, providing up to 15% improvement compared to other configurations. We achieved an ingestion rate of 120k EPS(Events per second), which translates to 8.58MB/sec. The input execution rate was approximately 33 inputs per second with each input ingesting an average of 370k events. Please Note: That above results were received on fully dedicated DB Connect environment:

• separate instance for Splunk with DB Connect ( no other apps/connectors installed)
• MySQL databases were also running on separate instances

Each environment should be treated individually and performance may be impacted by many factors. Once performance issues are noticed, this is the signal to check the environment configuration and how resource usage looks and it is also time to think about scaling.

Please find detailed results here [[Benchmarking & Scalability Metrics]]

The benchmarking focused on testing three DBX versions: 3.9.0, 3.10.0, and 3.17.2, while the scalability tests involved running DBX version 3.18.0 on three configurations (low, medium, and high) to assess their performance across different number of inputs (500, 1500, and 5000) running on a given environment. The goal of benchmarking and scalability tests was to measure DBX's performance in terms of CPU and memory utilisation, event ingestion rates, and input handling capacity across different environments. By identifying the best-performing configuration, we aim to determine the optimal specifications for running DBX at scale, ensuring high efficiency and reliability under heavy loads. DBX has been tested on m5.4xlarge instance type with following specifications:

• CPU: 16 vCPUs)
• Memory: 64 GiB
• OS: Ubuntu 22.04
• Storage: 200 GiB

If you are still experiencing performance issues, or want to receive feedback tailored to your setup, you have the following options:

• Post a request to the community on Splunk Answers.
• Contact Splunk Support.