Upgrade pre-9.2 deployment servers

Note: Starting with version 10.0, we have updated the following product names:
  • "deployment server" is now called "agent management"
  • "forwarder management" is now called "agent management"

Additionally, with version 10.0, the "deployment client" has been renamed into "agent". However, the name "deployment client" remains present in commands, attributes and .conf file names throughout this documentation and when interacting with agent management.

Please note that while the names have changed, the underlying functionality remains the same. Additionally new capabilities are introduced.

Aspects of the deployment server have been significantly enhanced in Splunk Enterprise version 9.2 to improve performance and manageability. In addition, the improvements enable multiple deployment servers to coordinate their activities in a deployment server cluster, as described in Implement an agent management cluster.

Because of these architectural improvements, deployment server upgrades that span the 9.2 release automatically undergo a number of changes to implement these improvements. For standalone deployment servers, no action is necessary on the part of the user beyond the normal process of upgrading a Splunk Enterprise instance, as the changes are implemented in an entirely automatic fashion. In addition, agents, including pre-9.2 clients, continue to operate seamlessly with the updated deployment servers.

However, if you examine the standalone deployment server directories, you will notice some differences. In particular, there is a new system-generated app, etc/apps/SplunkDeploymentServerConfig, which contains configuration files necessary to the proper functioning of the deployment server. Do not alter this directory or its files in any way. Note that this app is not a deployment app and so does not reside in etc/deployment-apps.

In addition, the system places new configurations in savedsearches.conf and macros.conf. Do not edit these system-generated configurations.

There are also some new logs generated by the deployment server in response to client phone home activities. These are placed in the client_events directory, new in version 9.2.

Possible issues with upgrade

Data not appearing in the interface

This problem can occur in Splunk Enterprise 9.2 or higher if your deployment server forwards its internal logs to a standalone indexer or to the peer nodes of an indexer cluster. This issue can occur after an upgrade or in a new installation of 9.2 or higher. To rectify, add these settings to outputs.conf on the deployment server:

CODE 
[indexAndForward]
index = true
selectiveIndexing = true

If you add these settings post-upgrade or post-installation, you might need to restart the deployment server.

Indexers require new internal deployment server indexes

The deployment server uses several internal indexes new in version 9.2. These indexes are included in all indexers at the 9.2 level and higher, but if you try to forward data from those indexes to a pre-9.2 indexer, problems can result.

If you forward data to your indexer tier, create these new internal deployment server indexes in indexes.conf on any pre-9.2 indexers in your environment:

CODE 
[_dsphonehome]
[_dsclient]
[_dsappevent]

If the indexers are at version 9.2 or higher, they are already configured with those indexes.

Note: If you add those indexes to peer nodes on an indexer cluster, be sure to set repFactor = auto, as you must for all peer node indexes.

Data does not appear when forwarded through an intermediate forwarder

This problem can occur if your deployment server forwards its internal index data through an intermediate forwarder to a standalone indexer or to the peer nodes of an indexer cluster. To rectify, add this setting to outputs.conf on the intermediate forwarder:

CODE 
[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

If you specify the configuration within a deployment app and use the deployment server to deploy the app to the affected intermediate forwarders, you can later uninstall the app when the intermediate forwarders are upgraded to a future release that incorporates the update.

Splunk Cloud with an on-premises deployment server

If you use Splunk Cloud and run your deployment server (DS) on-premises, you must configure your DS to keep _ds* indexes local and avoid forwarding them to Splunk Cloud. The _ds* indexes do not exist in Splunk Cloud, and forwarding them causes issues with Cloud automation.

Configure selective indexing

To ensure that _ds* logs are indexed locally while other internal logs are still forwarded to Splunk Cloud, add the following configuration to outputs.conf on your deployment server:

CODE 
[indexAndForward]
index = true
selectiveIndexing = true

When selectiveIndexing is set to true, only events with _INDEX_AND_FORWARD_ROUTING = local configured in inputs.conf are indexed locally. This setting is already partially configured by default in the etc/apps/SplunkDeploymentServerConfig app, which ships with the Splunk Enterprise installation.

Recommended practice

Forward logs from all your instances to the indexing layer.

On your DS instances, additionally enable the [indexAndForward] configuration above. This ensures that regular internal logs are forwarded to Splunk Cloud, while _ds* logs remain local to the DS instance.

Because _ds* indexes are internal indexes, no additional licensing is required for local indexing on the DS instance.