Bootstrap unique certificates for Splunk forwarders

Configure each universal forwarder that connects to your Splunk Cloud Platform deployment to automatically replace the shared client certificate in its forwarder bundle with a unique certificate on first startup, reducing the security risk of a shared certificate across all of your forwarders.

Universal forwarder (UF) bundles that you download from Splunk Cloud Platform include a single shared client certificate. Every forwarder that you provision from the same bundle uses that certificate to authenticate its Splunk-to-Splunk (S2S) connections to your indexers. Because all of these forwarders share the same credential, a security breach that exposes one forwarder's certificate can compromise the authentication of every other forwarder in your environment. The Secure Forwarder Bootstrap feature eliminates this risk by automatically provisioning each forwarder with a unique client certificate the first time it connects, without requiring any manual certificate management.

How forwarders receive unique certificates

When forwarders connect to a Splunk Cloud Platform deployment using mutually authenticated TLS (mTLS), each forwarder presents a client certificate during the TLS handshake. Forwarders that the same universal forwarder bundle provisions share an identical client certificate, which means that a compromised forwarder certificate can expose all of your forwarders. The Secure Forwarder Bootstrap feature resolves this by adding a bootstrap marker to the shared certificate in the UF bundle.

Note: Until the bootstrap workflow completes and the forwarder installs its unique certificate, the forwarder continues to use the shared bundle certificate. You are responsible for the security of the shared certificate during this period.

The Secure Forwarder Bootstrap marker takes the form of an X.509 standard Subject Alternative Name (SAN) DNS entry, called the bootstrap SAN, with a value of bootstrap.forwarders.stack_name.env.splunkcloud.com. When a forwarder with a true value for the autoCertRotation setting in its outputs.conf configuration file starts and detects this bootstrap SAN in its current certificate, it initiates the certificate rotation workflow through its existing S2S connection to the indexer. The Splunk Cloud Platform provisions a unique certificate and delivers it to the forwarder. When the forwarder installs the unique certificate, it uses the unique certificate for all S2S connections without requiring a restart.

This feature applies to universal forwarders that run on Windows and Linux and connect directly to indexers on a Splunk Cloud Platform deployment. Forwarders that route traffic through a heavy forwarder cannot use this feature. It does not apply to on-premises indexer deployments.

Note: Splunk Cloud Platform supports this feature only on deployments that Amazon Web Services (AWS) hosts. Contact Splunk Support to confirm whether your deployment supports the Secure Forwarder Bootstrap feature.

When the forwarder obtains a unique certificate, it does not repeat the bootstrap workflow on subsequent restarts because the unique certificate does not contain the bootstrap SAN. The forwarder automatically renews the certificate based on the value you set in the certRotationThresholdPct setting in the outputs.conf configuration file.

Prerequisites for certificate bootstrapping

Before you begin, confirm that your environment meets the following requirements:

  • Your Splunk Cloud Platform deployment runs version 10.6 or higher.

  • Your universal forwarder instances run version 10.6 or higher.

  • Splunk Support has turned on the Secure Forwarder Bootstrap feature for your deployment.

  • You have downloaded and deployed the updated UF bundle from Splunk Web after Splunk Support confirms that the update is complete. This bundle contains a certificate with a special X.509 Subject Alternative Name (SAN) DNS entry, called the bootstrap SAN, that marks the certificate for automatic replacement on first startup.

  • The forwarder machine runs Windows or Linux and connects directly to indexers, not through a heavy forwarder.

Procedure for bootstrapping unique certificates for Splunk forwarders

Configure universal forwarders connected to Splunk Cloud Platform to replace their shared bundle certificate automatically with a unique client certificate on first startup.

Review the prerequisites for certificate bootstrapping earlier in this topic.

Follow these steps to configure the Secure Forwarder Bootstrap feature on your universal forwarder instances.

  1. Contact Splunk Support to submit a support case requesting that they turn on the Secure Forwarder Bootstrap feature for your deployment.

    Splunk Support does not turn on this feature by default, and you cannot turn it on yourself. Splunk Support must update the deployment configuration and perform a certificate rotation to provision the bootstrap certificate into the UF bundle.

    After Splunk Support confirms that the certificate rotation is complete, download the updated splunkclouduf.spl bundle from Splunk Web and deploy it to your UF instances. All forwarders continue to use the shared certificate until each forwarder completes the bootstrap workflow.

    CAUTION: If you upgrade UF instances to version 10.6 as part of this rollout, do not back up and restore $SPLUNK_HOME/etc/apps/100_stackname/default/100_stackname_server.pem during the upgrade. Restoring this file places the old certificate without the bootstrap SAN back on the forwarder and prevents the bootstrap workflow from running. After upgrading, install the fresh bundle from Splunk Web instead. On all subsequent upgrades after the initial bootstrap is complete, you can resume the standard backup and restore practice for $SPLUNK_HOME/etc/.
  2. Optional: If you manage forwarder configuration through a deployment server (DS), configure the excludeFromUpdate setting in the serverclass.conf configuration file on the DS before you continue.

    The DS calculates its bundle checksum from .conf files only, so autoCertRotation can update a forwarder's certificate without the DS detecting a change. However, if any .conf file in the DS-managed app changes later, the DS pushes a new bundle that includes the original certificate, overwriting the unique certificate that the bootstrap workflow placed on the forwarder.

    Set excludeFromUpdate to prevent the DS from overwriting the certificate file during bundle pushes:

    CODE 
    [serverClass:<group>:app:100_<stackname>]
excludeFromUpdate = $app_root$/default/100_<stackname>_server.pem
  3. Open the outputs.conf configuration file on the forwarder machine in a text editor.

    The default file location depends on your operating system:

    • Linux: $SPLUNK_HOME/etc/system/local/outputs.conf

    • Windows: %SPLUNK_HOME%\etc\system\local\outputs.conf

    If the file does not exist, create it.

  4. In the [tcpout:target_group] stanza, give the autoCertRotation setting a value of true.
    CODE 
    [tcpout:<target_group>]
autoCertRotation = true
    Note: Place the autoCertRotation setting in a named [tcpout:target_group] stanza. Do not set it at the global [tcpout] level in the outputs.conf configuration file. If you define autoCertRotation in more than one named stanza, the bootstrap workflow runs only for the alphabetically first group name.
  5. Optional: Give the certRotationThresholdPct setting a whole-number percentage value to control when the renewal workflow starts for subsequent rotations.

    The forwarder begins the renewal workflow when the remaining certificate lifetime falls below this percentage of the certificate's total validity period. The default value is 15. Increase this value if you want renewal to begin earlier. For example, a value of 20% starts renewal when 20% of the certificate lifetime remains.

    CODE 
    [tcpout:<target_group>]
autoCertRotation = true
certRotationThresholdPct = 20
  6. Save the file.
  7. Restart the forwarder or perform a hot reload to apply the configuration change without a full restart.

    Linux:

    CODE 
    $SPLUNK_HOME/bin/splunk restart

    Windows (run as Administrator):

    CODE 
    "%SPLUNK_HOME%\bin\splunk.exe" restart

    To perform a hot reload without restarting the forwarder, run:

    CODE 
    splunk reload outputs

    For more information about when configuration changes require a restart and when you can reload without one, see When to restart Splunk Enterprise after a configuration file change.

    A restart or hot reload causes the forwarder to reconstruct its tcpout groups, which triggers the bootstrap workflow. The forwarder detects the bootstrap SAN in its current certificate and initiates the certificate rotation workflow through its existing Splunk-to-Splunk (S2S) connection to the indexer. The Splunk Cloud Platform provisions a unique certificate and delivers it to the forwarder, which installs it in place. The forwarder does not require an additional restart.

Each forwarder now uses a unique certificate for all S2S connections to Splunk Cloud Platform.

To monitor progress and verify certificate installation, see Monitor bootstrap progress and verify unique certificate installation later in this topic.

For large UF fleets, deploy the updated bundle and turn on the autoCertRotation setting in batches to avoid overwhelming Splunk Cloud Platform with simultaneous certificate signing requests. If Splunk Cloud Platform throttles a request, S2S forwarding continues uninterrupted on the existing certificate and the forwarder retries automatically. The next retry occurs between 18 and 30 hours after the throttled request. Monitor bootstrap progress in splunkd.log on each forwarder to confirm that the forwarder has replaced the bootstrap certificate with a unique certificate.

Monitor bootstrap progress and verify unique certificate installation

Review forwarder log messages to confirm that certificate bootstrapping started and that the forwarder installed its unique client certificate successfully.

To monitor bootstrapping and verify that the forwarder installed its unique certificate, review the splunkd.log log file on each forwarder for entries on the S2SCertUpdater log channel. This channel logs at the INFO logging level by default, and you do not need to change the level to see these log entries.

The log file locations depend on your operating system:

  • Linux: $SPLUNK_HOME/var/log/splunk/splunkd.log

  • Windows: %SPLUNK_HOME%\var\log\splunk\splunkd.log

Confirm that the forwarder initiated the bootstrapping process

When the forwarder detects the bootstrap SAN and initiates the rotation process, the following lines appear together in the log. The CertificateData line shows the certificate details, including the bootstrap SAN in the subjectAltName field. The S2SCertUpdater line immediately following it confirms that the rotation process started:

CODE 
INFO  CertificateData [...] - channel=CertificateSigningRequest, subject="CN=*.stack.splunkcloud.com", subjectAltName="DNS:*.stack.splunkcloud.com, DNS:bootstrap.forwarders.stack.splunkcloud.com", serial=..., notValidBefore=..., notValidAfter=..., issuer="...", sha256-fingerprint=...
INFO  S2SCertUpdater [...] - Starting rotation process for group=group_name
INFO  S2SCertUpdater [...] - Will check cert rotation in 23 hours, 12 minutes, 40 seconds

The scheduling line appears immediately because the forwarder schedules the next check immediately after it queues the certificate signing request. The response from the indexer arrives later.

Confirm that the forwarder installed its unique certificate

There is no explicit success log line at the default INFO logging level. If bootstrapping succeeds, the following line appears with no subsequent ERROR entries on the PKI or S2SCertUpdater log channels:

CODE 
INFO  S2SCertUpdater [...] - Start processing cert rotation for response origin=
Note: The origin= field is blank in all cases. Splunk designed this behavior intentionally.

After this line, a successful bootstrap results in a tcpout configuration reload in the log and no errors. If you see no ERROR entries on the PKI or S2SCertUpdater log channels after this line, the forwarder installed its unique certificate successfully.

Troubleshoot a failed bootstrap

Use forwarder log messages to identify certificate bootstrapping failures and turn on DEBUG logging for additional details about the certificate request.

If the bootstrap fails, the PKI channel logs an error first, followed immediately by an error on S2SCertUpdater:

CODE 
ERROR PKI [...] - The certificate signing response contains errors. Errors=..., Json response=...
ERROR S2SCertUpdater [...] - Error processing cert rotation for response origin=

The PKI error gives the specific reason. You might see the following additional failure patterns:

CODE 
ERROR S2SCertUpdater [...] - Error processing cert rotation for responseo origin =
WARN  S2SCertUpdater [...] - Received invalid resposne with non-empty origin: value
INFO  S2SCertUpdater [...] - S2S certificate request interrupted by tcpout reload; retrying soon. exception text
INFO  S2SCertUpdater [...] - No feature enabled TcpOutput group to send the certificate request
Note: Currently, the typos "responseo" and "resposne" appear verbatim in the source code. Search for these exact strings when looking for these log entries.

The first pattern is the empty-payload case. The third pattern ("S2S certificate request interrupted") indicates that a tcpout reload interrupted the request; the forwarder retries automatically. The fourth pattern ("No feature enabled") means that a named [tcpout:target_group] stanza lacks the autoCertRotation setting.

Add the following line to the $SPLUNK_HOME/etc/log-local.cfg configuration file on the forwarder to raise the logging level on the PKI channel to DEBUG:

CODE 
category.PKI=DEBUG

Restart splunkd.

With DEBUG logging turned on, the bootstrap SAN detection and certificate signing request (CSR) creation steps appear in the log before the CertificateData line.

Turn off certificate bootstrapping

Contact Splunk Support and deploy an updated forwarder bundle to turn off certificate bootstrapping and restore the standard shared certificate on your forwarders.

Use this procedure to turn off certificate bootstrapping for your forwarders.

  1. Contact Splunk Support to submit a support case requesting that they turn off the Secure Forwarder Bootstrap feature for your deployment.
  2. After Splunk Support performs a certificate rotation to regenerate the UF bundle without the bootstrap SAN, download the updated splunkclouduf.spl bundle from Splunk Web.
  3. Deploy the updated bundle to your UF instances to restore the standard shared certificate.