When provisioning agent management, note the following:

• If it will be deploying to more than 50 clients, the agent management must run on a dedicated Splunk Enterprise instance. The instance cannot double as an indexer or search head.
• Do not host a distributed management console, which is essentially a search head, on an agent management with more than 50 clients.
• Do not host a search head cluster deployer on an agent management with more than 50 clients.
• If the agent management has less than 50 clients, you can co-locate the agent management on an indexer or search head, including a distributed management console.
• Because of high CPU and memory usage during the app download process, it is recommended that the instance reside on a dedicated host machine.

The guidance here is based on testing done using the following set-up:

• Agent management running on a dedicated bare metal 64-bit Linux system with 12GB RAM and 12 cores.
• Deployment across a high-speed LAN network. On higher latency networks, the time to deploy will be longer.
• An agent phone home interval of 60 seconds.
• A set of apps totaling 50MB in size. This is considered a relatively large size to deploy, because in most situations you would deploy a set of apps only once, followed by incremental updates.

To estimate the total time it will take to deploy 50MB to all your agents (assuming the agent management hardware discussed above), you can use this formula:

T = 0.0075 * C + 2

where T is the maximum time to deploy in minutes, and C is the number of agents.

For example, it would take a maximum of 9.5 minutes to deploy 50MB of apps to 1000 agents.

The guidance here is based on testing done using the following set-up:

• An agent management running on a dedicated 64-bit Windows Server 2008 virtual machine, with 12GB RAM and 8 cores.
• Deployment across a high-speed LAN network. On higher latency networks, the time to deploy will be longer.
• An agent phone home interval of 60 seconds.
• A set of apps totaling 50MB in size. This is considered a relatively large size to deploy, because in most situations you would deploy a set of apps only once, followed by incremental updates.

To estimate the total time it will take to deploy 50MB to all your agents (assuming the agent management hardware discussed above), you can use this formula:

T = 0.04 * C + 8

where T is the maximum time to deploy in minutes, and C is the number of agents. This formula is valid for deployments up to 3000 agents.

If you are deploying to more than 2000 agents, you might experience significantly improved performance by increasing the phone home interval to five minutes (300 seconds).

The dedicatedIoThreads setting in server.conf specifies the number of threads dedicated to handling HTTP I/O requests. You do not usually need to change this setting from its default of "auto".

However, for an instance running as a dedicated agent management and servicing a large number of agents (>10,000), the best practice is to set dedicatedIoThreads to a value of CPU cores/2, or CPU cores-1, in the case of a very active agent management. For example, in the case of an agent management with 8 CPU cores, set dedicatedIoThreads to 4 or 7.