Configure workload pools
This topic discusses how to configure workload pools on a single instance. For information on how to extend this to distributed distributed deployments, see Configure workload management on distributed deployments.
Follow these steps to configure workload management on a single instance:
Run preflight checks
A set of system checks runs automatically to determine if your underlying Linux operating system is set up properly for creating workload pools. If any preflight check fails, review the error messages to identify the Linux configuration issues you must fix before you can configure workload pools.
You can run preflight checks manually using the CLI or REST API.
Workload management runs the following preflight checks:
Name | Mitigation |
---|---|
OS | Operating system must be Linux. Workload management is not currently supported on Windows OS. |
Cgroup version | Cgroup must be version 1. Workload management does not support pre-cgroup or cgroup version 2 Linux kernels. |
CPU Splunk base directory check |
CPU Splunk base directory For The For non-systemd, the base directory is workload_pool_base_dir_name defined in workload_pools.conf The default value is splunk . See Configure Linux systems not running systemd for workload management.
|
CPU Splunk base directory permissions | CPU Splunk base directory Splunkd.service requires read and write permissions. For systemd , permissions must be set for non-root user in the Splunkd.service unit file. See Configure Linux systemd for workload management. For non-systemd, use |
Memory Splunk base directory check | Memory Splunk base directory Splunkd.service is missing. For systemd , the base directory is /sys/fs/cgroup/memory/system.slice/<unit_file_name> . See Configure Linux systemd for workload management. For non-systemd, the base directory is |
Memory Splunk base directory permissions | Memory Splunk base directory Splunkd.service requires read and write permissions. For systemd , permissions must be set for non-root user in the Splunkd.service unit file. See Configure Linux systemd for workload management. For non-systemd, use |
Unit file check | The unit file Splunkd.service is missing. The unit file is located under /etc/systemd/system with the name <SPLUNK_SERVER_NAME>.service . SPLUNK_SERVER_NAME is set in splunk-launch.conf . See Configure Linux systemd for workload management.
|
Delegate property set to true | The Delegate property in the unit file must be set to true . |
Splunk launched under systemd | splunkd is running as a systemd service. In the unit file, the Restart property must be set to always . The ExecStart property must include _internal_launch_under_systemd . |
For more information on unit file properties, see systemd unit file properties.
For more information on how to set up Linux for workload management, see Set up Linux for workload management.
Run preflight checks in Splunk Web
- Click Settings > Workload Management. The Linux preflight checks run automatically. If all preflight checks pass, the workload management UI opens, and you can proceed to configure workload management.
- If any preflight check fails, a page appears showing the check results. Review the error messages and fix the specified Linux configuration issues.
- After fixing the issues, click Rerun preflight checks.
Run preflight checks using the CLI
To run preflight checks for workload management using the CLI:
- Log in to your Linux machine.
- Run the following CLI command:
./splunk check workload-config
Here is an example of the output from this command:
Workload Management Preflight Checks failed. Fix the following issues: CPU Splunk base directory Splunkd.service requires read and write permissions. CPU Splunk base directory Splunkd.service is missing. The 'Delegate' property in the unit file must be set to 'true'. Restart Splunk then rerun preflight checks. In the unit file, the 'Restart' property must be set to 'always'. The 'ExecStart' property must include '_internal_launch_under_systemd'. Make sure the up-to-date unit file is loaded. Memory Splunk base directory Splunkd.service requires read and write permissions. Memory Splunk base directory Splunkd.service is missing. Unit file Splunkd.service is missing. Restart Splunk then rerun preflight checks.
Run preflight checks using REST
Send a GET request to:
workloads/config/preflight-checks
For endpoint details, see workloads/config/preflight-checks in the REST API Reference Manual.
Configure workload categories
Workload categories are created by default and each workload category has its own default CPU and memory resource allocation. You can modify the resource allocation for each category.
The workloads that run in each category are:
- Search: All searches run in this category. You can create several pools under the search category.
- Ingest: Indexing and Splunk core processes run in this category.
- Misc: Scripted inputs and modular inputs run in this category.
You can edit workload categories using Splunk Web, the CLI, or REST.
Edit workload categories using Splunk Web
To edit the resource allocation for a workload category in Splunk Web, do the following:
- In Splunk Web, click Settings > Workload Management. The workload management UI opens.
- Click the All Categories tile.
- Click Edit under the specific category.
- Specify the resource allocation:
Field Action CPU Weight Specify the total CPU weight available for pools in this category. Unused CPU cycles are automatically shared with workloads in other categories. Memory Limit % Specify the maximum percentage of Memory available for pools in this category. - Click Submit.
For more information, see Resource allocation in workload management.
Edit workload categories using the CLI
To edit the resource allocation for a workload category, run the following CLI command:
./splunk edit workload-category <category> [-cpu_weight <number> -mem_weight <number>]
where <category> is search, ingest, or misc.
To list workload category information, run the following CLI command:
./splunk list workload-category
Edit workload categories using REST
To edit the resource allocation for a workload category, send a POST request to the following endpoint:
workloads/categories
For endpoint details, see workloads/categories in the REST API Reference Manual.
Create workload pools
A workload pool is defined under a workload category. Each workload pool gets a specified percentage of the total CPU and memory resources that are available to that category.
You must create, at a minimum, these two workload pools:
- Default pool in the Search category: Searches that are not explicitly placed in a workload pool by a workload rule are assigned to this pool.
- Default pool in the Ingest category: Indexing and other non-search Splunk core processes are assigned to this pool.
You can optionally create a default pool in the Misc category. Scripted and modular inputs run in this pool by default. If you do not create a default pool in the Misc category, scripted and modular inputs run in the default pool in the ingest category.
You can create and edit workload pools using Splunk Web, the CLI, or REST API. Workload pool configurations are stored in workload_pools.conf
. See View workload_pools.conf.
Create a workload pool in Splunk Web
- In Splunk Web, click Settings > Workload Management.
- Click Add Workload Pool.
- Specify the following fields:
Field Action Pool Category Select a workload category based on the type of process the pool will run (search, ingest, or misc). See Configure workload categories. Name Specify the name of the workload pool. Valid characters are alphanumeric and underscore only. CPU Weight The fraction of total available CPU for this pool. Unused CPU cycles are automatically shared with workloads in other pools. Memory Limit % The maximum percentage of total available memory for this pool. Default Pool Toggle the switch to make this pool the default pool for the selected category. - Click Submit. The workload pool appears in the Workload Management UI.
For more information, see Resource allocation in workload management.
Create a workload pool using the CLI
Run the following CLI command:
./splunk add workload-pool <pool_name> [-category <search/ingest/misc> -cpu_weight <number> -mem_weight <number> -default_category_pool <true|false>]
Create a workload pool using REST
Send a POST request to the following endpoint:
workloads/pools
For endpoint details, see workloads/pools in the REST API Reference Manual.
View workload_pools.conf
When you create a workload pool, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_pools.conf
.
workload_pools.conf
defines the cpu and memory resource allocation for workload categories (search, ingest, and misc) and the individual workload pools created under those categories. For example:
[general]
default_pool = pool_1
ingest_pool = pool_3
enabled = 0
[workload_category:search]
cpu_weight = 70
mem_weight = 70
[workload_category:ingest]
cpu_weight = 20
mem_weight = 20
[workload_category:misc]
cpu_weight = 10
mem_weight = 10
[workload_pool:pool_1]
cpu_weight = 70
mem_weight = 70
category = search
default_category_pool = 1
[workload_pool:pool_2]
cpu_weight = 30
mem_weight = 30
category = search
default_category_pool = 0
[workload_pool:pool_3]
cpu_weight = 100
mem_weight = 100
category = ingest
default_category_pool = 1
[workload_pool:pool_4]
cpu_weight = 100
mem_weight = 100
category = misc
default_category_pool = 1
For more information, see workload_pools.conf.
workload_pools.conf
files in more than one app context. Having identical workload_pools.conf
stanzas in multiple app contexts can cause workload management enable/disable functions to fail and cause other issues.Delete workload pools
You can delete any workload pool under a category, except for the default category pool. If you try to delete the default category pool an error message appears. You can delete workload pools using Splunk Web, the CLI, or REST API.
To delete a workload pool using the CLI:
./splunk remove workload-pool <pool_name>
workload_pools.conf
reloads or Splunk Enterprise restarts.