View configurations installed on your forwarders

You can quickly view the configuration installed on your forwarders, without the need to log in to the forwarders, or run btool, by using the Effective Configuration add-on.

Overview

The Effective Configuration add-on saves time and eliminates the need to contact other teams, helping you access the information you need in seconds instead of days.

btool is a command-line tool used in Splunk Enterprise to simulate the merging of configuration files and report on the merged settings. For more information about btool, see Use btool to troubleshoot configurations.

Note: btool shows the configuration saved on disk and not the one in memory.

Using the Effective Configuration add-on also allows you to view the actual configuration used on agents, including all parameter changes. You can view the current effective configuration on selected agents and check parameter values set in the following files:

  • inputs.conf
  • outputs.conf
  • deploymentclient.conf
  • server.conf

To view the effective configuration that is currently used on the forwarders, complete the following tasks:

  1. Meet all prerequisites.
  2. Install the Effective Configuration add-on.
  3. (Optional) Reduce the size of the Effective Configuration add-on.
  4. Verify that the Effective Configuration add-on is installed.

Security

The connection is initiated from the agent side, using the same management port for communication. Requests are authenticated with the pass4SymmKey and can be further secured using TLS settings. The Effective Configuration add-on reuses the same TLS settings as the forwarder, considering all settings specified under the [sslConfig] stanza in the server.conf file.

The minimum supported SSL version for communication between the agent and the agent management is TLS 1.0.

Limitations

If you're managing a large number of agents, avoid installing the Effective Configuration add-on on all agents at the same time. To reduce load, set maxConcurrentDownloads = 17 in the serverclass.conf file. This slightly slows down the process of installation and sending out the Effective Configuration add-on, but the configuration is delivered shortly after the add-on is installed on the agent. This value is based on the results from add-on tests.

The scalability setup of agent management

To use the agent management in a clustered environment with the scalability feature introduced in version 9.2, it is essential to configure a shared drive for var/lib/effective-configuration. This setup is similar to the configuration of other shared directories. For more information, see Implement a deployment server cluster.

Cleanup process

Currently, there is no strict limit on the storage of effective configurations files on disc. However, a pre-established soft limit manages storage effectively. When this limit is exceeded, the system automatically removes outdated effective configurations to free up space. An outdated effective configuration means that an agent hasn't sent a single phonehome for more than 10 times its average phonehome interval.

For example, if the average phonehome interval of an agent is 1 minute, the storage limit is exceeded and if no phonehome is sent for 10 minutes, the effective configuration for that agent is removed.

The following setting is used in the agent_management.conf file: 

[effective_configuration]
max_size = 16
cleanup_threshold = 6144
cleanup_schedule = 0 3 * * *

By default, the cleanup process is scheduled to run daily at 3 AM local server time. It runs only if the total size of saved effective configurations exceeds 6,144 MB. Additionally, it removes effective configurations only for agents that have been inactive for a significant period of time.

Note: In the cluster mode, only 1 agent management can have a configured cleanup process.

To turn off the cleanup process, set cleanup_schedule = disabled .

The following calculations outline the total effective configuration size that you can use to set the cleanup threshold based on the fleet size:

  • Minimum required disk storage: 125 MB for every 1,000 agents
  • Recommended disk storage: 250 MB for every 1,000 agents

Prerequisites

To use the Effective Configuration add-on, fulfill the following requirements:

  • Use Splunk Enterprise version 10.0 and higher.
  • Use agents in the following versions of Splunk Enterprise:
    • Version 8.0 and higher for the universal forwarder - version 8.0 and higher on every supported platform except Solaris Sparc.
    • Version 8.0 and higher on heavy forwarders, agent managements, and search heads.
  • Set up 1 or more agents, such as universal forwarders. For more information, see Plan a deployment.
  • Set up pass4SymmKey.

Set up pass4SymmKey

  1. You can set pass4SymmKey for the [deployment] stanza in the server.conf file for both agent management and agents. For more information, see server.conf.

    You can also set pass4SymmKey on agents by deploying an application from the agent management. The application sets the pass4SymmKey and restarts the agent management. For more information, see Secure Splunk Enterprise services with pass4SymmKey.

  2. Restart the agents and agent management by using the CLI command ./bin/splunk restart.
  3. Verify if agents can communicate with phonehome:
    1. Log in to your Splunk platform.
    2. Go to Settings and select Agent management under the Distributed environment section.
    3. On the Agent Management page, go to the Forwarders tab.
    4. Verify that the agent status is OK to confirm proper connectivity.

Install the Effective Configuration add-on

By default, the Effective Configuration add-on is not installed on agents. To install the Effective Configuration add-on on agents, you need to download the add-on from Splunkbase and then deploy it to agents. For more information about deploying apps, see Deploy apps to clients.

Note: Before you begin, make sure that you use agent management version 10.0.0 or higher.
  1. Download Splunk_TA_effective_configuration from Splunkbase.
  2. Untar a .tgz file into the etc/deployment-apps directory.

    You have to create an app directory for the agent management to distribute this add-on. The default location is $SPLUNK_HOME/etc/deployment-apps, but this is configurable through the repositoryLocation attribute in serverclass.conf. For more information, see Create the app directories.

  3. Log in to Splunk Enterprise, go to Settings and select Agent management under the Distributed environment section.
  4. On the Agent Management page, go to the Applications tab. In the list of applications, you can see Splunk_TA_effective_configuration. Its deployment status is set to Not Deployed.
  5. To deploy the Effective Configuration add-on, go to the Server Classes tab.
  6. You can create a new server class or select an existing one. From the list of server class names, select the one you want to use. You need to add the Effective Configuration add-on to this server class to install it on agents.
    Note: You have to add the add-on to at least 1 server class in order to be able to edit this application.
  7. On the Server classes page, go to the Applications tab and select Edit applications.
  8. On the Edit applications page, in the Unassigned Applications section, select Splunk_TA_effective_configuration.
  9. Select the right arrow to move the selected application to the Assigned Applications section.
  10. Select Save. The application is sent when the agent phonehome is sent from the agent. When the application is saved, the Applications tab displays.
  11. In the table, select Splunk_TA_effective_configuration.
  12. In the Details tab, update these settings:
    1. Select the Enable application option.
    2. Turn on the Restart agent toggle.
  13. (Optional) You can add forwarders to the server class and wait for the add-on installation to complete:
    1. Go to the Forwarder Management page and then go to the Server Classes tab.
    2. Select a server class to which you want to add forwarders.
    3. Go to the Forwarders tab and select Edit forwarders. For more details about possible actions in this page, see Specify clients.
    4. Specify the clients you want in the server class, and then select Save.
  14. After the installation is complete, typically within 1 phonehome period, you can access the effective configuration used on forwarders. For more information, see View configurations installed on forwarders.

(Optional) Reduce the size of the Effective Configuration add-on

After downloading the Splunk_TA_effective_configuration add-on, you can reduce the size of this add-on by removing directories that are not used for the target platform. Depending on your needs, you can remove the following directories:
  • aix*
  • darwin*
  • freebsd*
  • linux*
  • solaris*
  • windows*

For example, if you use this add-on only on linux_arm64 platform, you can remove the following directories: windows_x86_64, windows_x86, solaris_amd64, linux_x86_64, linux_s390x, linux_ppc64le, freebsd_amd64,darwin_x86_64, darwin_arm64 and aix_ppc64 directories.

Verify that the Effective Configuration add-on is installed

Note: The add-on installation is tied to the phonehome interval. A higher phoneHomeIntervalInSecs value means the configuration change, such as an application change, it takes up to that amount of time to apply.
  1. To check if the Effective Configuration add-on is installed, on the Server classes page, go to the Forwarders tab and select a forwarder.
  2. On the Forwarders page, go to the Applications tab and check that the status of the Splunk_TA_effective_configuration add-on is set to Deployed.

View configurations installed on forwarders

When the Effective Configuration add-on is installed on forwarders in your organization, you can view the parameter values that are installed in .conf files on forwarders. You can also check the default values for parameters. If you prefer to view the configuration in a different editor, you can download the Effective Configuration add-on files.

  1. Log in to your Splunk platform.
  2. Go to Settings and select Forwarder management under the Distributed environment section.
  3. On the Forwarder Management page, select a forwarder on which the effective configuration is installed.
  4. On the Forwarder page, select the Effective Configuration tab. Under this tab, you can view the configuration you received from the agent. The 4 tabs indicate the inputs.conf, outputs.conf, deploymentclient.conf, and server.conf files.
  5. Select a tab to view the configuration in a given .conf file.
  6. To view the whole configuration with default values, turn on the Show default values toggle.
    • The default values are indicated by the gray color.
    • The values that were updated are indicated by the white color.

    Turn off the Show default values toggle to view the updated values that are installed on the forwarder.

  7. Use the search bar to find a value that you want to check.
  8. Check the time when the effective configuration was generated and uploaded. Hover over the time in the Generated and Uploaded fields.
    • The Generated field shows the date and time when btool was run on the agent to generate effective configuration.
    • The Uploaded field shows the date and time when the server saved the effective configuration received from the agent.

Download configurations installed on forwarders

You can download the configuration files that are installed on a given forwarder and open the files in a text editor of your choice.

  1. Log into your Splunk platform.
  2. Go to Settings and select Forwarder management under the Distributed environment section.
  3. On the Forwarder Management page, select a forwarder on which the Effective Configuration add-on is installed.
  4. On the Forwarder page, select the Effective Configuration tab.
  5. Select Download effective configuration. The files with configuration are downloaded as a zip folder. It contains 4 .conf files in text format: inputs.conf, outputs.conf, deploymentclient.conf, server.conf.
  6. You can open and view the files in a text editor of your choice.