Network routing and ingress for Splunk POD

Splunk POD provides access to cluster components using IP-based routing by default.

Access Splunk POD cluster components using the default IP-based routing or by manually configuring name-based routing. You can also set up the HTTP Event Collector (HEC) to ingest data over HTTP and establish secure S2S connections for universal forwarders.

IP based routing

By default, the cluster provides ingress using the IP address of any worker node.

Each node runs a load balancer that routes traffic to the appropriate service within the cluster. Any worker node IP address provides equivalent access to cluster services.

Port and URL Mapping

In this configuration, the Splunk Service API shares the same port as the Web UI.

Note: The splunkd management port 8089 is not exposed externally.
Component Port TLS URL Example
SHC 1 UI 443 Yes https://<WORKER_IP>
SHC 1 API 443 Yes https://<WORKER_IP>/services
SHC 2 UI 8100 Yes https://<WORKER_IP>:8100
SHC2 API 8100 Yes https://<WORKER_IP>:8100/services
Standalone SH 1 UI 8000 Yes https://<WORKER_IP>:8000
Standalone SH 1 API 8000 Yes https://<WORKER_IP>:8000/services
Standalone SH 2 UI 8001 Yes https://<WORKER_IP>:8001
Standalone SH 2 API 8001 Yes https://<WORKER_IP>:8001/services
HEC (HTTP Event Collector) 443 Yes https://<WORKER_IP>/services/collector
Cluster Manager 1443 Yes https://<WORKER_IP>:1443
License Manager 2443 Yes https://<WORKER_IP>:2443
Monitoring Console 3443 Yes https://<WORKER_IP>:3443
S2S Forwarding 9997 No tcp://<WORKER_IP>:9997
Perses 3000 Yes https://<WORKER_IP>:3000

To retrieve a list of current worker node IPs, run the following installer command:

CODE 
./kubernetes-installer-standalone -static.cluster cluster-config.yaml -status.workers

Configure name-based routing

Configure DNS and TLS for name-based ingress routing to Splunk POD cluster components.

Splunk POD provides name-based ingress routes for accessing cluster components, in addition to IP-based routes. Name-based routing maps DNS names to Splunk POD services, replacing IP addresses and port numbers with URLs that contain identifiable component names.

Examples:

  • https://core-shc.pod.example.com for a search head cluster named "core-shc"
  • https://manager.pod.example.com for the cluster manager

Search head routes use the tier names defined in the static cluster configuration file. For example, a search head cluster named core-shc is accessible at https://core-shc.pod.example.com. Deployer routes append -deployer to the search head cluster name, for example https://core-shc-deployer.pod.example.com.

Named routes for Splunk components default to the Splunk Web UI on port 443. Append :8089 to the URL to access the Splunk REST API.

Name-based routing requires the following:

  • A wildcard DNS A record mapping to all worker node IP addresses.
  • A TLS wildcard certificate with a domain name matching the DNS configuration.

  • A certificate stanza in the static configuration file referencing the TLS certificate.

Named ingress routes

The following table lists the named ingress routes for Splunk POD components. All examples use the domain pod.example.com.

Component Port TLS URL example Notes
Search Head Cluster "my-shc" UI 443 Yes https://my-shc.pod.example.com Uses the SHC name from the static configuration file.
Search Head Cluster "my-shc" API 8089 Yes https://my-shc.pod.example.com:8089 Uses the SHC name from the static configuration file.
Standalone Search Head "my-sh" UI 443 Yes https://my-sh.pod.example.com Uses the standalone search head name from the static configuration file.
Standalone Search Head "my-sh" API 8089 Yes https://my-sh.pod.example.com:8089 Uses the standalone search head name from the static configuration file.
Cluster Manager UI 443 Yes https://manager.pod.example.com
Cluster Manager API 8089 Yes https://manager.pod.example.com:8089
License Manager UI 443 Yes https://license.pod.example.com
License Manager API 8089 Yes https://license.pod.example.com:8089
Monitoring Console UI 443 Yes https://console.pod.example.com Splunk metrics
Monitoring Console API 8089 Yes https://console.pod.example.com:8089
Deployer for SHC "my-shc" UI 443 Yes https://my-shc-deployer.pod.example.com Uses the SHC name + "-deployer".
Deployer for SHC "my-shc" API 8089 Yes https://my-shc-deployer.pod.example.com:8089 Uses the SHC name + "-deployer".
Prometheus UI 443 Yes https://prometheus.pod.example.com Kubernetes metrics
Perses UI 443 Yes https://perses.pod.example.com Kubernetes metrics visualizations
HTTP Event Collector 443 Yes https://hec.pod.example.com/services/collector
S2S Forwarding 9997 No s2s.pod.example.com:9997

Configure DNS

Create a single wildcard DNS A record (IPv4) that maps to every worker node IP address. An A record can map a single DNS name to multiple IP addresses, and client applications such as browsers receive the full list from the DNS server.

The IP addresses in the DNS record should match the full set of worker IPs from the Splunk POD static cluster configuration file. Although name-based routing functions with a single worker IP, it is best practice to include every worker IP for resiliency.

Update the DNS record whenever worker node IP addresses change, including when a worker node is replaced. Named routes continue to function with a single working IP address in the DNS record.

Note: The Kubernetes Installer does not configure DNS. Each environment has its own DNS configuration process.

DNS A record example:

  • Name: *.pod.example.com
  • Record type: A
  • TTL (seconds): 60
  • Resource records:
    • 10.237.5.90
    • 10.237.5.149
    • 10.237.5.70

Configure TLS certificate and private key

Name-based routes function without a TLS certificate, but browsers will display TLS warnings. To enable trusted HTTPS access, specify a TLS wildcard certificate and its associated private key in the static cluster configuration file. Both files must be in PEM format.

Static cluster configuration file example:

CODE 
certificate:
  ingress:
    certificate: ./fullchain.pem
    privateKey: ./privkey.pem

The TLS certificate file can include a full certificate chain, with each certificate listed in series. The certificate should use a wildcard domain matching the DNS configuration, for example *.pod.example.com.

Certificate file example (PEM):

CODE 
-----BEGIN CERTIFICATE-----
<server certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<intermediate CA certificate>
-----END CERTIFICATE-----

Private key file example (PEM):

CODE 
-----BEGIN PRIVATE KEY-----
<private key contents>
-----END PRIVATE KEY-----

Certificate renewal: You are responsible for renewing TLS certificates before they expire. The installer pre-flight checks flag expired certificates and certificates within 30 days of expiration. To renew a certificate, update the PEM files and redeploy:

CODE 
./kubernetes-installer-standalone --static.cluster <path-to-static-config> -deploy

Configure HTTP Event Collector (HEC)

HEC is the primary method for data ingestion into Splunk Enterprise over HTTPS, targeting the indexer tier on port 443.

Use the HTTP Event Collector (HEC) to ingest data into Splunk Enterprise over HTTPS. HEC targets the indexer tier on port 443.
  1. Configure HEC data sources using the endpoint format for the cluster’s routing method:

    IP-Based routing:

    Specify multiple worker node IPs in the HEC configuration.

    https://<WORKER_IP_1>:443/services/collector https://<WORKER_IP_2>:443/services/collector

    Name-based routing:

    Use the designated DNS name provisioned during IngressRoute setup:

    https://hec.pod.example.com/services/collector

  2. Verify HEC connectivity.

    Send an HTTP POST request to confirm that you can access and send data to the HEC endpoint. Use the default HEC token provided with your Splunk POD credentials in the authorization header.

    JSON 
    curl -k "https://<WORKER_IP>:443/services/collector" \
-H "Authorization: Splunk <YOUR_HEC_TOKEN>" \
-d '{"event": "Connection Test", "sourcetype": "manual"}'

Configure S2S connections

The Splunk-to-Splunk (S2S) protocol provides secure data ingestion from Splunk Universal Forwarders (UF).

The Splunk-to-Splunk (S2S) protocol provides secure data ingestion from Splunk Universal Forwarders (UF).

S2S is available for TLS traffic over TCP port 9997, targeting the indexer tier. The Kubernetes installer does not manage or interact with forwarders. Configure forwarders manually or use agent management on the bastion node.

To establish a secure S2S connection, install matching security apps on both the indexer tier (within the cluster) and on forwarders (outside the cluster):

  1. Configure indexers:

    Deploy an app that contains a properly configured inputs.conf file and the required certificates. See Configure Splunk indexing and forwarding to use TLS certificates.

    To install the app, use the clustermanager.apps.cluster scope in the cluster configuration file.

  2. Configure forwarders:

    Prepare a forwarder bundle with an outputs.conf file and certificates. See Configure forwarding with outputs.conf. Deploy this bundle to the forwarders manually or by using agent management on the bastion node.

    While S2S traffic can reach every indexer pod through a single host:port pair, you must provide multiple addresses in the outputs.conf server list for proper load balancing. For UF version 9.4.1 and higher, set forcedTimeBasedAutoLB = true in the outputs.conf to ensure that Layer 4 Ingress correctly balances traffic across the indexers.

    For IP-based routing, list all worker node IPs in the server list:

    CODE 
    [tcpout]
defaultGroup = splunk_pod_indexers

[tcpout:splunk_pod_indexers]
server = 10.237.5.177:9997, 10.237.5.239:9997, 10.237.5.146:9997
useACK = true
forcedTimeBasedAutoLB = true
sslRootCAPath = /opt/splunkforwarder/etc/auth/ca.pem
sslCertPath = /opt/splunkforwarder/etc/auth/client.pem
sslPassword = <encrypted_password>
sslVerifyServerCert = true

    For name-based routing, list multiple DNS names:

    CODE 
    [tcpout]
defaultGroup = splunk_pod_indexers

[tcpout:splunk_pod_indexers]
server = inputs001.pod.example.com:9997, inputs002.pod.example.com:9997
useACK = true
forcedTimeBasedAutoLB = true
sslRootCAPath = /opt/splunkforwarder/etc/auth/ca.pem
sslCertPath = /opt/splunkforwarder/etc/auth/client.pem
sslPassword = <encrypted_password>
sslVerifyServerCert = true

    Additional resources: