Bootstrap unique certificates for Splunk forwarders

Configure each Universal Forwarder that connects to your Splunk Cloud Platform deployment to automatically replace the shared client certificate in its UF bundle with a unique certificate on first startup, reducing the security risk of a shared certificate across all of your forwarders.

Universal Forwarder bundles that you download from Splunk Cloud Platform include a single shared client certificate. Every forwarder that you provision from the same bundle uses that certificate to authenticate its Splunk-to-Splunk (S2S) connections to your indexers. Because all of these forwarders share the same credential, a security breach that exposes one forwarder's certificate can compromise the authentication of every other forwarder in your environment. The Secure Forwarder Bootstrap feature eliminates this risk by automatically provisioning each forwarder with a unique client certificate the first time it connects, without requiring any manual certificate management.

How the bootstrap workflow works

When forwarders connect to a Splunk Cloud Platform deployment using mutually authenticated TLS (mTLS), each forwarder presents a client certificate during the TLS handshake. Forwarders that the same Universal Forwarder (UF) bundle provisions share an identical client certificate, which means that a compromised forwarder certificate can expose all of your forwarders. The Secure Forwarder Bootstrap feature resolves this by adding a bootstrap marker to the shared certificate in the UF bundle.

The Secure Forwarder Bootstrap marker takes the form of an X.509 Subject Alternative Name (SAN) extension—called the bootstrap SAN—with a value of bootstrap.forwarders.stack_name.env.splunkcloud.com. When a forwarder starts with the autoCertRotation setting in the outputs.conf configuration file set to true and detects this bootstrap SAN in its current certificate, it automatically requests a unique certificate from the Splunk Cloud Platform certificate authority (CA). When the forwarder installs the unique certificate, it uses the unique certificate for all S2S connections without requiring a restart.

This feature applies to Universal Forwarders that run on Windows and Linux and connect directly to indexers on a Splunk Cloud Platform deployment. Forwarders that route traffic through a heavy forwarder cannot use this feature. It does not apply to on-premises indexer deployments.

When the forwarder obtains a unique certificate, it does not repeat the bootstrap workflow on subsequent restarts because the unique certificate does not contain the bootstrap SAN. The forwarder automatically renews the certificate based on the value you set in the certRotationThresholdPct setting in the outputs.conf configuration file.

Prerequisites to bootstrapping unique certificates for Splunk forwarders

Before you begin, confirm that your environment meets the following requirements:

  • Your Splunk Cloud Platform deployment runs version 10.6 or higher.
  • Splunk Support has turned on the Secure Forwarder Bootstrap feature for your deployment.
  • You have downloaded and deployed the updated UF bundle from Splunk Web after Splunk Support confirms that the update is complete. This bundle contains a certificate with a special X.509 Subject Alternative Name (SAN) extension—called the bootstrap SAN—that marks the certificate for automatic replacement on first startup.
  • The forwarder machine runs Windows or Linux and connects directly to indexers, not through a heavy forwarder.
  • The forwarder machine can reach the Splunk Cloud Platform certificate authority (CA) over HTTPS. The CA is the service that issues unique certificates to your forwarders.

Procedure for bootstrapping unique certificates for Splunk forwarders

Follow these steps to configure the Secure Forwarder Bootstrap feature on your Universal Forwarder instances.

  1. Contact Splunk Support to submit a support case requesting that they turn on the Secure Forwarder Bootstrap feature for your deployment.

    Splunk Support does not turn on this feature by default, and you cannot turn it on yourself. Splunk Support must update the deployment configuration and perform a certificate rotation to provision the bootstrap certificate into the UF bundle.

    After Splunk Support confirms that the certificate rotation is complete, download the updated splunkclouduf.spl bundle from Splunk Web and deploy it to your UF instances. All forwarders continue to use the shared certificate until each forwarder completes the bootstrap workflow.

    CAUTION: If you upgrade UF instances to version 10.6 as part of this rollout, do not back up and restore $SPLUNK_HOME/etc/apps/100_stackname/default/100_stackname_server.pem during the upgrade. Restoring this file places the old certificate without the bootstrap SAN back on the forwarder and prevents the bootstrap workflow from running. After upgrading, install the fresh bundle from Splunk Web instead. On all subsequent upgrades after the initial bootstrap is complete, you can resume the standard backup and restore practice for $SPLUNK_HOME/etc/.
  2. If you manage forwarder configuration through a deployment server (DS), configure the excludeFromUpdate setting in the serverclass.conf configuration file on the DS before you continue.

    The DS calculates its bundle checksum from .conf files only, so autoCertRotation can update a forwarder's certificate without the DS detecting a change. However, if any .conf file in the DS-managed app changes later, the DS pushes a new bundle that includes the original certificate, overwriting the unique certificate that the bootstrap workflow placed on the forwarder.

    Set excludeFromUpdate to prevent the DS from overwriting the certificate file during bundle pushes:

    CODE 
    [serverClass:group:app:100_stackname]
excludeFromUpdate = $app_root$/default/100_stackname_server.pem
  3. Open the outputs.conf configuration file on the forwarder machine in a text editor.

    The default file location depends on your operating system:

    • Linux: $SPLUNK_HOME/etc/system/local/outputs.conf
    • Windows: %SPLUNK_HOME%\etc\system\local\outputs.conf

    If the file does not exist, create it.

  4. In the [tcpout:target_group] stanza, give the autoCertRotation setting a value of true.
    CODE 
    [tcpout:target_group]
autoCertRotation = true
    Note: Place the autoCertRotation setting in a named [tcpout:target_group] stanza. Do not set it at the global [tcpout] level in the outputs.conf configuration file. If you define autoCertRotation in more than one named stanza, the bootstrap workflow runs only for the alphabetically first group name.
  5. Optional: Give the certRotationThresholdPct setting a whole-number percentage value to control when the renewal workflow starts for subsequent rotations.

    The forwarder begins the renewal workflow when the remaining certificate lifetime falls below this percentage of the certificate's total validity period. The default value is 15. Increase this value if you want renewal to begin earlier. For example, a value of 20 starts renewal when 20% of the certificate lifetime remains.

    CODE 
    [tcpout:target_group]
autoCertRotation = true
certRotationThresholdPct = 20
  6. Save the file.
  7. Restart the forwarder or perform a hot reload to apply the configuration change without a full restart.

    Linux:

    CODE 
    $SPLUNK_HOME/bin/splunk restart

    Windows (run as Administrator):

    CODE 
    "%SPLUNK_HOME%\bin\splunk.exe" restart

    A restart or hot reload causes the forwarder to reconstruct its tcpout groups, which triggers the bootstrap workflow. The forwarder detects the bootstrap SAN in its current certificate, contacts the Splunk Cloud Platform CA through the Secrets Management Agent (SMA), receives a unique certificate, and replaces the bootstrap certificate in place. The forwarder does not require an additional restart.

Each forwarder now uses a unique certificate for all S2S connections to Splunk Cloud Platform. To confirm that the bootstrap completed successfully, search splunkd.log on each forwarder for entries in the PKI log channel that contain the cert_request_id field. The following log message confirms that the bootstrap certificate was replaced with a unique certificate:

CODE 
cert_request_id=uuid, Certificate rotation complete. certpath=path

Log file locations:

  • Linux: $SPLUNK_HOME/var/log/splunk/splunkd.log
  • Windows: %SPLUNK_HOME%\var\log\splunk\splunkd.log

To roll back the bootstrap workflow:

  1. Contact Splunk Support to submit a support case requesting that they turn off the Secure Forwarder Bootstrap feature for your deployment.
  2. After Splunk Support performs a certificate rotation to regenerate the UF bundle without the bootstrap SAN, download the updated splunkclouduf.spl bundle from Splunk Web.
  3. Deploy the updated bundle to your UF instances to restore the standard shared certificate.

For large UF fleets, deploy the updated bundle and turn on the autoCertRotation setting in batches to avoid overwhelming the CA with simultaneous certificate signing requests. If the CA throttles a request, S2S forwarding continues uninterrupted on the existing certificate and the forwarder retries automatically. The next retry occurs between 18 and 30 hours after the throttled request. Monitor bootstrap progress in splunkd.log on each forwarder to confirm that the forwarder has replaced the bootstrap certificate with a unique certificate.