Steps for securing your Splunk Enterprise deployment with TLS

The general workflow to secure your Splunk platform deployment with TLS follows:

Understand the basics of and learn the terms for TLS. This helps you understand the concepts of configuring TLS on your Splunk platform instances.
Decide how you want to secure your Splunk platform deployment. This determines how much securing work you actually do.
- If you use Splunk Cloud Platform, your Splunk Cloud Platform infrastructure already has certificates that protect it. Splunk provides and maintains these certificates, including the certificates that protect connection between forwarders and SCP.. You can consider the following options for the data collection and forwarding infrastructure that you manage and that sends data to your Splunk Cloud Platform instance.
- You can choose to secure communications between Splunk Web and your browser, or Splunk Web and the Splunk platform deployment
- You can also secure communications between individual Splunk platform instances. This is similar to securing Splunk Web, but has a slightly different procedure
- You can secure both of these types of communication. This provides the best level of security but takes additional time and requires a better understanding of your Splunk platform deployment and its position in the network.
Obtain the TLS certificates that you need to secure the deployment in the way you want
- You can get the certificates from a third party, or
- You can create the certificates yourself
Verify that the certificates are valid
Install the certificates on each Splunk platform instance
Configure each Splunk platform instance to use the certificates
If necessary, configure your domain name service (DNS) registry to account for the information that the certificates contain
Test and troubleshoot

Note: This topic is a high-level workflow for obtaining and installing certificates for Splunk Platform instances that you manage. It does not contain specifics on configurations, certificate store locations, or certificate composition requirements. See the links in each section that follows for specific details on that section, or read through the topic and proceed to the Next Steps section to continue into each section to begin installing and configuring certificates.

1. Understand the basics of and learn the terms for TLS

Before you attempt to configure the Splunk platform to use transport layer security, understand the basic concepts and learn the terms of TLS.

You must implement TLS correctly in your Splunk platform deployment to ensure that the deployment operates at its most secure. Understanding key terms helps you more quickly put all the files and configurations into the correct places.

The following table describes the terms you will need to understand to be able to configure the Splunk platform to use TLS.


Term	Definition
Private Key	One of two keys that the TLS protocol uses to authenticate and secure network connections between Splunk components and services. The private key is the key that only you, as its owner, knows. The key is represented by a file that is linked to another file which represents the public key. All TLS certificates use private keys.
Public Key	One of two keys that the TLS protocol uses to authenticate and secure network connections between Splunk components and services. The public key is the key that is known to the outside world. The key is represented by a file that is linked to another file which represents the private key.
Rivest, Shamir, and Adelman (RSA)	The encryption technology that is used for private keys, as required by the Splunk platform. Private keys that you use on the Splunk platform must be in this format.
Certificate Signing Request	A certificate signing request (CSR) is a message that you, as an applicant, send to a certificate authority for the purposes of creating a certificate. The message contains details about one or more servers that you want to use the certificate with.
Certificate Authority	A certificate authority (CA) is an authority that validates the identity of entities through the issuance of certificates. CAs control the root and intermediate certificates that applicants use to sign CSRs into certificates. Typically a CA is a third party vendor, but can also be an internal team or even you.
Certificate	A file that lets entities that communicate using TLS to safely establish connections and encrypt data between one another. A certificate lets these entities prove to each other that they are who they say they are. Also called a server certificate or leaf certificate.
Leaf Certificate	A certificate that represents one or more servers. The TLS connection uses the leaf certificate for encryption. Also called a "server certificate" or "client certificate". See More information on leaf certificates.
Intermediate Certificate	A certificate that a CA typically uses to sign leaf, or client/server certificates.
Root Certificate	A certificate that a CA typically uses to sign intermediate certificates. Also called a "CA certificate" or an "issuer certificate".
Full Certificate Chain	The full set of certificates that a Splunk platform node requires to establish a secure connection to another Splunk platform node over TLS.
Privacy Enhanced Mail (PEM)	The data format that certificates must be in for the Splunk platform to use the certificates. See More information on the PEM format.
Mutually authenticated transport layer security (mTLS)	A TLS configuration where both the client who initiates the connection and the server that receives the connection validate each other's certificates prior to the establishment of that connection.

More information on leaf certificates

There are several kinds of leaf certificates, based on the instance type that uses the certificates.

Server certificates are leaf certificates that reside on machines that support incoming connections from clients. Typically, server certificates fall into three types:

Individual server certificates are configured to match one server name only.
Multi-host server certificates are configured to match more than one server name. Each server name in a multi-host server certificate is spelled out in its configuration.
Wildcard server certificates are configured to match all server names in a specific DNS domain.

Client certificates are leaf certificates that reside on machines that support making outgoing connections to servers. A server can host both a server certificate for connections from incoming clients and a client certificate that it uses to connect to other servers.

Splunk platform nodes such as forwarders use client certificates when the servers that they connect to require the client to present such a certificate to complete setup of the secure connection through a configuration. The server validates the certificate that the client presents before letting the connection continue. The client certificate does not necessarily need to represent the client specifically - it merely needs to have the correct configuration for the server to accept it.

More information on the PEM format

All digital certificates that the Splunk platform uses must be in privacy-enhanced mail (PEM) format. If they do not come in this format, you must convert them. When you create the certificates yourself, the instructions here generate them in PEM format automatically.

For details on how to convert certificates that are not in this format, see How to prepare TLS certificates for use with the Splunk platform.

2. Decide how you want to secure your Splunk platform deployment

This step in the workflow is the most challenging because it is the most open-ended and changes based on your specific situation. As a Splunk administrator, it's your responsibility to ensure that the data that your Splunk Enterprise or Splunk Cloud Platform supporting infrastructure is protected. How you protect that data depends on the topology of your network, the number of instances in your Splunk platform infrastructure, and the sensitivity of your data. While Splunk best practice is to secure all parts of your Splunk platform infrastructure, your use case might limit the ability to do so.

When you decide how to secure your Splunk platform infrastructure, you should take into account the following:

Does any of your infrastructure use the internet to communicate with other parts of your infrastructure, even to connect with Splunk Cloud Platform?
Similarly, do you have external users that connect to your infrastructure over the internet to use the Splunk platform?
Do you need to securely connect your forwarding tier infrastructure to Splunk Cloud Platform?

Answers to these questions determine what you secure in your infrastructure, and where. As a reminder, Splunk best practice is to secure all contact points of your Splunk platform deployment wherever you can, even if you protect that deployment with a firewall.

There are two basic areas in Splunk platform infrastructure that you can secure:

Secure Inter-Splunk communications

You can use TLS certificates to secure communications that involve communication between individual nodes in your Splunk infrastructure. This involves almost any kind of node that does not use the Splunk Web interface. It's also known as "Splunk-to-Splunk" communication. This type of exchange involves communication between things like:

Forwarders and indexers
Search heads and search peers
Any kind of clustered instance, like an indexer or search head cluster, or a search head deployer
Agent management and agents
License managers and license peers
The Splunk CLI and any Splunk instance you want to connect to using the CLI
The Splunk App Key Value Store service
Python modules within the Splunk platform installation

Secure Splunk Web communications

You secure Splunk Web communications similarly to, but slightly different than, you do communications between Splunk platform instances. There is a configuration for connecting your browser with Splunk Web, and another for connecting Splunk Web with search heads and search head clusters.

3. Obtain the certificates that you need to secure your Splunk platform deployment

After you determine how you want to protect the infrastructure, the next step is to get the TLS certificates that perform this protection.

With Splunk Cloud Platform, it's straightforward. Splunk protects the contact points in your Splunk Cloud Platform instance with TLS technology, updates the certificates for you as needed, and provides the Universal Forwarder Credentials Package which you can use to connect forwarders to send data to Splunk Cloud Platform.

On Splunk Enterprise and for connections between instances in your Splunk Cloud Platform-supporting infrastructure, this work is your responsibility. You must obtain, install, and configure certificates to protect data communication between Splunk platform instances.

Obtain certificates from a third party

You can contact a third party certificate authority (CA) and obtain a signed certificate from them to install into your Splunk platform infrastructure. This is the most secure option that is available because third party CAs are known entities.

Caveats to using this method include a delay in receiving the certificates and likely a cost for obtaining them, depending on the certificate authority you use and how you obtain the certificates.

See the following topics for specifics:

Create and sign certificates yourself

In this scenario, you are the certificate authority.

If you have experience creating and signing TLS certificates, you can generate, sign, and install the certificates into your Splunk platform deployment, then configure the deployment to use the certificates. You can use any method you are comfortable with to generate the certificates, as long as the certificates you generate are in the correct file format and meet the requirements for the level of security you want to use them.

See the following topic for specifics:

How to create and sign your own TLS certificates

4. Verify that the certificates are valid

After you get the certificates, and before you install them, you need to verify that the certificates you received are valid.

The certificates must be in the correct file format, usually the privacy-enhanced mail (PEM) format. If they aren't, you have to convert them to this format
The certificate must contain the correct information as the certificate authority provided it
The certificate must contain the information that you provided as part of setting up the certificate. For example, it must contain information about the domains, or Common Names, that you want the certificate to protect. If it is a wildcard certificate, which is one that protects multiple domains, it must also contain any subdomains that you want protected.
It must be valid for the dates that you want the protection to be in force. You can't extend a certificate that's expired, instead you must replace it with a new one. Splunk Enterprise won't run with a certificate that hasn't yet come into force.

See the following topic for specifics:

How to prepare TLS certificates for use with the Splunk platform

5. Install the certificates on each Splunk platform instance

After you obtain the certificates, you can install and configure the instances to see and use them.

Generally, you can install a certificate directly into the certificate store on your Splunk platform instance by creating a directory within that certificate store and placing the files you downloaded there.

Depending on the complexity of your deployment, and the interaction with any existing certificates you might have in the deployment, you might need to perform additional steps to make the certificates work in the deployment:

You might need to convert the format of your certificate if it isn't in the required format.
You might need to concatenate certificates, especially if your environment uses multiple certificates or certificate chains as part of a securement strategy that supersedes your Splunk platform deployment. Splunk platform instances must see a complete certificate chain to operate properly.

See the following topics for specifics:

6. Configure each Splunk platform instance to use the certificates

After you have installed the certificates into your Splunk platform installation, your Splunk platform deployment needs to know where to find and use them. In nearly all cases, you perform this configuration using Splunk configuration files.

The configuration files you use, and the location within each configuration file where you do the configuration, depends on several factors:

The Splunk platform instance function: indexer, search head, cluster node, and so on
The Splunk service whose contact points you want to protect. For example, the Splunk daemon, the CLI, App Key Value Store, Python modules, and so on

When you perform configurations, you follow the same rules for configuring the location and usage of certificates that you would when you set up other parts of a configuration file. Configuration file precedence and other rules apply.

See the following topics for specifics:

7. Configure your Domain Name Services (DNS) registry properly

DNS is an integral part of securing your Splunk platform infrastructure.

While you can obtain a certificate for one or more IP addresses from a public CA, the addresses must be public, and you must demonstrate that you own them. If you can't, they won't issue that kind of certificate to you. They also won't issue certificates for private IP addresses.

You must ensure your DNS is properly configured, even if you host the deployment on a completely private network. All machines that run Splunk deployment infrastructure must have properly configured DNS.

This is because secure communications using certificates must leverage either a Common Name or a Subject Alternative Name that is present in the certificates that you obtain and deploy. While it's possible to use IP addresses, even private ones, in certificates, using fully-qualified domain names (FQDN) and having DNS perform proper resolution eases the certificate configuration process significantly.

Review your DNS configuration. Ensure that DNS servers run the latest versions of DNS software.
Confirm that DNS records and references are properly configured.
Test DNS on each Splunk platform instance. At a minimum, the machine should be able to resolve its own fully qualified domain name and the names of machines to which it connects to an IP address.

8. Test and troubleshoot your certificates after installation

After you install the certificates, you can test the security of your Splunk deployment by reviewing the Splunk daemon logs on each of the instances to confirm that there are no errors in the configuration.

Start the Splunk platform instances.
Review the Splunk daemon logs, either by performing a search within Splunk Web or by reviewing the log file on the instance directly.
Search for errors that contain references to TLS and SSL. If you don't find any, your Splunk platform instances are properly configured.
If you do find errors, review the logs to determine what caused the errors, then review your certificates to ensure they meet the requirements for your specific application, taking into account the types of certificates, the domains that the certificates protect, interaction with other certificates, and other factors.

For more information about testing and troubleshooting your TLS configuration, see Test and troubleshoot TLS connections.

Post-setup configuration tasks

After you configure and test your certificates and TLS connections, you can further bolster the security of your network by performing the following optional tasks.

Configure certificate host name validation

Version 8.2.2202 and higher of Splunk Cloud Platform and 9.0.0 and higher of Splunk Enterprise let you configure your Splunk platform instances so that they verify the host names in certificates that they receive when they connect to other Splunk platform instances. Certificate requirements, in addition to certificate host name validation, work to further improve security within your Splunk platform deployments.

See Enable TLS certificate host name validation for additional information and procedures.

Configure mutually authenticated TLS (mTLS)

Version 10.0.0 and higher of Splunk Enterprise let you configure your Splunk platform instances to use mutually authenticated transport layer security (mTLS) for specific types of connections between Splunk platform services. mTLS provides increased security by requiring that both sides of a TLS connection validate one another through digital certificates before letting network traffic pass through the connections.

See Configure mutually authenticated transport layer security (mTLS) on the Splunk platform for additional information and procedures.

Next Steps

If you have a Splunk Cloud Platform deployment with external infrastructure that forwards data to it, see the following topic to configure the universal forwarder credentials package on that forwarding infrastructure:

Install and configure the Splunk Cloud Platform universal forwarder credentials package

If you have a Splunk Enterprise deployment, read on to understand the next steps for securing the infrastructure with certificates.

Now that you understand the overall procedure for using TLS certificates with your Splunk platform deployment, you need some certificates to work with if you don't already have them. Choose from one of the following links for specific instructions on getting or creating the certificates.

If you want to obtain a certificate from a third-party CA to protect communication between Splunk platform instances, see How to obtain certificates from a third party for inter-Splunk communication.
If you want to obtain a certificate from a third-party CA to protect communications that involve Splunk Web, see How to obtain certificates from a third party for Splunk Web.
If you want to create and sign your own certificates to protect any part of Splunk Enterprise including Splunk Web, see How to create and sign your own certificates for use with the Splunk platform.
If you have already obtained certificates and just need to know how to configure them to use with the Splunk platform, see How to prepare TLS certificates for use with the Splunk platform.