Network routing and ingress for Splunk POD

Configure name-based routing that uses DNS records to route traffic to services.

1. Prepare TLS secrets
   1. Encode certificate and key. Use base64 to encode both the certificate (tls.crt) and key (tls.key) and copy the results into the following secret YAML file.
      CODECopy

      cat tls.crt | base64 -w 0 > tls.crt.b64 cat tls.key | base64 -w 0 > tls.key.b64

      cat tls.crt | base64 -w 0 > tls.crt.b64
      cat tls.key | base64 -w 0 > tls.key.b64

   2. Create secrets. Create a secret YAML file, splunk-tls.yml, on the bastion server using the base64-encoded string of the certificate chain and key file. Replace the placeholder values for tls.crt and tls.key with the encoded strings:
      CODECopy

      apiVersion: v1 kind: Secret metadata: name: splunk-all-tls namespace: splunk type: kubernetes.io/tls data: tls.crt: <BASE64_CERT_CONTENT> tls.key: <BASE64_KEY_CONTENT>

      apiVersion: v1
      kind: Secret
      metadata:
        name: splunk-all-tls
        namespace: splunk
      type: kubernetes.io/tls
      data:
        tls.crt: <BASE64_CERT_CONTENT>
        tls.key: <BASE64_KEY_CONTENT>

2. Create ingress routes

   Create an IngressRoute YAML file, splunk-ingress.yml to define the routing rules. Replace the Host sections with the FQDNs associated with the DNS records provisioned earlier. Each hostname must be unique, though you can modify them based on your naming conventions.

   Example configuration (splunk-ingress.yml):

   CODECopy

   apiVersion: traefik.io/v1alpha1 kind: IngressRoute metadata: name: ingress-https-all namespace: splunk spec: entryPoints: - https routes: # Search Head UI - kind: Rule match: Host(`search.pod.example.com`) services: - name: splunk-shc-search-head-service port: 8000 scheme: http serversTransport: selfsigned sticky: cookie: httpOnly: true # Search Head API (Routes management traffic to port 8089) - kind: Rule match: Host(`search.pod.example.com`) && PathRegexp(`^/(rpc|services|servicesNS|static)`) services: - name: splunk-shc-search-head-service port: 8089 scheme: https serversTransport: selfsigned # HEC Ingress - kind: Rule match: Host(`hec.pod.example.com`) priority: 20 services: - name: splunk-idx-indexer-service port: 8088 scheme: https serversTransport: selfsigned # License Manager - kind: Rule match: Host(`lm.pod.example.com`) services: - name: splunk-lm-license-manager-service port: 8000 scheme: https serversTransport: selfsigned # Cluster Manager - kind: Rule match: Host(`cm.pod.example.com`) services: - name: splunk-cm-cluster-manager-service port: 8000 scheme: https serversTransport: selfsigned # Monitoring Console - kind: Rule match: Host(`mc.pod.example.com`) services: - name: splunk-mc-monitoring-console-service port: 8000 scheme: https serversTransport: selfsigned tls: secretName: splunk-all-tls

   apiVersion: traefik.io/v1alpha1
   kind: IngressRoute
   metadata:
     name: ingress-https-all
     namespace: splunk
   spec:
     entryPoints:
       - https
     routes:
     # Search Head UI
     - kind: Rule
       match: Host(`search.pod.example.com`)
       services:
       - name: splunk-shc-search-head-service
         port: 8000
         scheme: http
         serversTransport: selfsigned
         sticky:
           cookie:
             httpOnly: true
     # Search Head API (Routes management traffic to port 8089)
     - kind: Rule
       match: Host(`search.pod.example.com`) && PathRegexp(`^/(rpc|services|servicesNS|static)`)
       services:
       - name: splunk-shc-search-head-service
         port: 8089
         scheme: https
         serversTransport: selfsigned
     # HEC Ingress
     - kind: Rule
       match: Host(`hec.pod.example.com`)
       priority: 20
       services:
       - name: splunk-idx-indexer-service
         port: 8088
         scheme: https
         serversTransport: selfsigned
     # License Manager
     - kind: Rule
       match: Host(`lm.pod.example.com`)
       services:
       - name: splunk-lm-license-manager-service
         port: 8000
         scheme: https
         serversTransport: selfsigned
     # Cluster Manager
     - kind: Rule
       match: Host(`cm.pod.example.com`)
       services:
       - name: splunk-cm-cluster-manager-service
         port: 8000
         scheme: https
         serversTransport: selfsigned
     # Monitoring Console
     - kind: Rule
       match: Host(`mc.pod.example.com`)
       services:
       - name: splunk-mc-monitoring-console-service
         port: 8000
         scheme: https
         serversTransport: selfsigned
     tls:
       secretName: splunk-all-tls

3. Apply configuration

   Apply the yaml configuration to the cluster to create the Kubernetes objects:

   1. Open the kubectl debug shell:
      CODECopy

      ./kubernetes-installer-standalone -static.cluster cluster-config.yaml -kubectl

      ./kubernetes-installer-standalone -static.cluster cluster-config.yaml -kubectl

   2. Apply the Secret and IngressRoute configurations:
      CODECopy

      kubectl apply -n splunk -f splunk-tls.yml kubectl apply -f splunk-ingress.yml

      kubectl apply -n splunk -f splunk-tls.yml
      kubectl apply -f splunk-ingress.yml

4. Verify configuration

   Verify that the ingressroute and secret configurations exist in the cluster.

   1. Verify Kubernetes resources:
      CODECopy

      kubectl get secrets -n splunk kubectl get ingressroutes -n splunk

      kubectl get secrets -n splunk
      kubectl get ingressroutes -n splunk

   2. Verify that the Web UI is accessible by navigating to https://search.pod.example.com.
      Note: If the page does not load immediately, allow time for DNS propagation across your local network.

HEC is the primary method for data ingestion into Splunk Enterprise over HTTPS, targeting the indexer tier on port 443.

1. Configure HEC data sources using the endpoint format for the cluster’s routing method:

   IP-Based routing:

   Specify multiple worker node IPs in the HEC configuration.

   https://<WORKER_IP_1>:443/services/collector https://<WORKER_IP_2>:443/services/collector

   Name-based routing:

   Use the designated DNS name provisioned during IngressRoute setup:

   https://hec.pod.example.com/services/collector

2. Verify HEC connectivity.

   Send an HTTP POST request to confirm that you can access and send data to the HEC endpoint. Use the default HEC token provided with your Splunk POD credentials in the authorization header.

   JSONCopy

   curl -k "https://<WORKER_IP>:443/services/collector" \ -H "Authorization: Splunk <YOUR_HEC_TOKEN>" \ -d '{"event": "Connection Test", "sourcetype": "manual"}'

   curl -k "https://<WORKER_IP>:443/services/collector" \
   -H "Authorization: Splunk <YOUR_HEC_TOKEN>" \
   -d '{"event": "Connection Test", "sourcetype": "manual"}'

The Splunk-to-Splunk (S2S) protocol provides secure data ingestion from Splunk Universal Forwarders (UF).

1. Configure indexers:

   Deploy an app that contains a properly configured inputs.conf file and the required certificates. See Configure Splunk indexing and forwarding to use TLS certificates.

   To install the app, use the clustermanager.apps.cluster scope in the cluster configuration file.

2. Configure forwarders:

   Prepare a forwarder bundle with an outputs.conf file and certificates. See Configure forwarding with outputs.conf. Deploy this bundle to the forwarders manually or by using agent management on the bastion node.

   While S2S traffic can reach every indexer pod through a single host:port pair, you must provide multiple addresses in the outputs.conf server list for proper load balancing. For UF version 9.4.1 and higher, set forcedTimeBasedAutoLB = true in the outputs.conf to ensure that Layer 4 Ingress correctly balances traffic across the indexers.

   For IP-based routing, list all worker node IPs in the server list:

   CODECopy

   [tcpout] defaultGroup = splunk_pod_indexers [tcpout:splunk_pod_indexers] server = 10.237.5.177:9997, 10.237.5.239:9997, 10.237.5.146:9997 useACK = true forcedTimeBasedAutoLB = true sslRootCAPath = /opt/splunkforwarder/etc/auth/ca.pem sslCertPath = /opt/splunkforwarder/etc/auth/client.pem sslPassword = <encrypted_password> sslVerifyServerCert = true

   [tcpout]
   defaultGroup = splunk_pod_indexers
   
   [tcpout:splunk_pod_indexers]
   server = 10.237.5.177:9997, 10.237.5.239:9997, 10.237.5.146:9997
   useACK = true
   forcedTimeBasedAutoLB = true
   sslRootCAPath = /opt/splunkforwarder/etc/auth/ca.pem
   sslCertPath = /opt/splunkforwarder/etc/auth/client.pem
   sslPassword = <encrypted_password>
   sslVerifyServerCert = true

   For name-based routing, list multiple DNS names:

   CODECopy

   [tcpout] defaultGroup = splunk_pod_indexers [tcpout:splunk_pod_indexers] server = inputs001.pod.example.com:9997, inputs002.pod.example.com:9997 useACK = true forcedTimeBasedAutoLB = true sslRootCAPath = /opt/splunkforwarder/etc/auth/ca.pem sslCertPath = /opt/splunkforwarder/etc/auth/client.pem sslPassword = <encrypted_password> sslVerifyServerCert = true

   [tcpout]
   defaultGroup = splunk_pod_indexers
   
   [tcpout:splunk_pod_indexers]
   server = inputs001.pod.example.com:9997, inputs002.pod.example.com:9997
   useACK = true
   forcedTimeBasedAutoLB = true
   sslRootCAPath = /opt/splunkforwarder/etc/auth/ca.pem
   sslCertPath = /opt/splunkforwarder/etc/auth/client.pem
   sslPassword = <encrypted_password>
   sslVerifyServerCert = true

   Additional resources:

   • About agent management
   • Configure Splunk indexing and forwarding to use TLS certificates
   • Configure forwarding with outputs.conf