Configure Ingress using Istio

Set up Istio ingress for Splunk Enterprise on Kubernetes.

Configure Istio as an ingress controller to route external requests to Splunk Enterprise services running in your Kubernetes cluster. Istio allows the cluster to receive requests from external sources and routes them to a desired destination within the cluster. Istio uses an Envoy proxy that allows for precise control over how data is routed to services by looking at attributes such as host name, URI, and HTTP headers. Through the use of destination rules, it also allows fine-grained control over how data is routed within services.

For instructions on how to install and configure Istio for your specific infrastructure, see the Istio getting started guide.

Most scenarios for Istio require the configuration of a Gateway and a Virtual Service. Familiarize yourself with the Istio Gateway and Istio Virtual Service.

Configure ingress for Splunk Web and HEC

You can configure Istio to provide direct access to Splunk Web.

Standalone configuration

  1. Create a Gateway to receive traffic on port 80:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: splunk-web
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: UI
      protocol: TCP
    hosts:
    - "splunk.example.com"
  2. Create a Virtual Service to route traffic to your service:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-web
spec:
  hosts:
  - "splunk.example.com"
  gateways:
  - "splunk-web"
  tcp:
  - match:
    - port: 80
    route:
    - destination:
        port:
          number: 8000
        host: splunk-standalone-standalone-service
  3. Get the External-IP for Istio:
    CODE 
    kubectl get svc -n istio-system
  4. Use a browser to connect to the External-IP to access Splunk Web:
    CODE 
    http://<LoadBalancer-External-IP>

Multiple hosts and HEC configuration

If your deployment has multiple hosts such as Search Heads and Cluster Manager, use this example to configure Splunk Web access and the HTTP Event Collector port. Follow the HEC documentation to learn how to create a HEC token and how to send data using HTTP.

  1. Create a Gateway for multiple hosts:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: splunk-web
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "splunk.example.com"
    - "deployer.splunk.example.com"
    - "cluster-manager.splunk.example.com"
    - "license-manager.splunk.example.com"
  2. Create a VirtualService for each component that you want to expose outside of Kubernetes:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-web
spec:
  hosts:
  - "splunk.example.com"
  gateways:
  - "splunk-web"
  http:
  - match:
    - uri:
        prefix: "/services/collector"
    route:
    - destination:
        port:
          number: 8088
        host: splunk-example-indexer-service
  - route:
    - destination:
        port:
          number: 8000
        host: splunk-example-search-head-service
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-deployer
spec:
  hosts:
  - "deployer.splunk.example.com"
  gateways:
  - "splunk-web"
  http:
  - route:
    - destination:
        port:
          number: 8000
        host: splunk-example-deployer-service
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-cluster-manager
spec:
  hosts:
  - "cluster-manager.splunk.example.com"
  gateways:
  - "splunk-web"
  http:
  - route:
    - destination:
        port:
          number: 8000
        host: splunk-example-cluster-manager-service
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-license-manager
spec:
  hosts:
  - "license-manager.splunk.example.com"
  gateways:
  - "splunk-web"
  http:
  - route:
    - destination:
        port:
          number: 8000
        host: splunk-example-license-manager-service
  3. Create a DestinationRule to ensure user sessions are sticky to specific search heads:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: splunk-search-head-rule
spec:
  host: splunk-example-search-head-service
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: SPLUNK_ISTIO_SESSION
          ttl: 3600s

If you are using HTTP Event Collector, modify your ingress-gateway service to listen for inbound TCP connections on port 8088:

JSON 
kubectl patch -n istio-system service istio-ingressgateway --patch '{"spec":{"ports":[{"name":"splunk-hec","port":8088,"protocol":"TCP"}]}}'

Configure ingress for Splunk Forwarder data

Configure the Istio Gateway and Virtual Service to enable inbound communications from Splunk Forwarders to the cluster.

  1. Create a Gateway:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: splunk-s2s
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 9997
      name: tcp-s2s
      protocol: TCP
    hosts:
    - "splunk.example.com"
  2. Create a Virtual Service:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-s2s
spec:
  hosts:
  - "splunk.example.com"
  gateways:
  - "splunk-s2s"
  tcp:
  - match:
    - port: 9997
    route:
    - destination:
        port:
          number: 9997
        host: splunk-example-indexer-service
  3. Modify your ingress-gateway service to listen for inbound TCP connections on port 9997:
    JSON 
    kubectl patch -n istio-system service istio-ingressgateway --patch '{"spec":{"ports":[{"name":"splunk-s2s","port":9997,"protocol":"TCP"}]}}'
  4. Use the External-IP from Istio in the forwarder's outputs.conf:
    CODE 
    kubectl get svc -n istio-system

Configure ingress for Splunk Forwarder data with TLS

Always use TLS encryption for your Splunk Enterprise endpoints.

End-to-end TLS

In this configuration, Istio passes the encrypted traffic to Splunk Enterprise without any termination. You must configure the TLS certificates on the forwarder as well as any Splunk Enterprise indexers, cluster peers, or standalone instances.

When using TLS for ingress, add an additional port for secure communication. By default, port 9997 handles non-encrypted traffic. You can use any other available port for secure communications.

The following example shows how to add port 9998 for a standalone instance:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: standalone
  labels:
    app: SplunkStandAlone
    type: Splunk
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  serviceTemplate:
    spec:
      ports:
      - name: tls-splunktest
        port: 9998
        protocol: TCP
        targetPort: 9998
  1. Modify your ingress-gateway Service to listen for S2S TCP connections on the new port (9998):
    JSON 
    kubectl patch -n istio-system service istio-ingressgateway --patch '{"spec":{"ports":[{"name":"splunk-tls","port":9998,"protocol":"TCP"}]}}'
  2. Create a Gateway with TLS Passthrough:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: splunk-s2s
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 9998
      name: tls-s2s
      protocol: TLS
    tls:
      mode: PASSTHROUGH
    hosts:
    - "*"
  3. Create a Virtual Service for TLS routing:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-s2s
spec:
  hosts:
  - "*"
  gateways:
  - "splunk-s2s"
  tls:
  - match:
    - port: 9998
      sniHosts:
      - "splunk.example.com"
    route:
    - destination:
        host: splunk-standalone-standalone-service
        port:
          number: 9998
    Note: This TLS example requires that outputs.conf on your forwarders includes the setting tlsHostname = splunk.example.com. Istio requires the TLS header to be defined so it knows which indexers to forward the traffic to. If this parameter is not defined, your forwarder connections fail.

    If you have only one indexer cluster as the destination for all S2S traffic, you can optionally replace splunk.example.com with the wildcard *. When you use this wildcard, you do not have to set the tlsHostname parameter in outputs.conf on your forwarders.

  4. Deploy an app to the standalone instance with the inputs.conf settings needed to open port 9998 and configure the relevant TLS settings. For details on app management using the Splunk Operator, see Using Apps for Splunk Configuration.

    Configure the forwarder's outputs.conf and the indexer's inputs.conf using the Configure Secure Forwarding documentation.

TLS Gateway termination

In this configuration, Istio terminates the encryption at the Gateway and forwards the decrypted traffic to Splunk Enterprise. You must configure the forwarder's outputs.conf for TLS, and configure the indexer's inputs.conf to accept non-encrypted traffic.

  1. Create a TLS secret with the certificates needed to decrypt traffic. These are the same commands used on your indexer to terminate TLS:
    CODE 
    kubectl create -n istio-system secret tls s2s-tls --key=<Path to private key> --cert=<Path to Indexer certificate>
  2. Create a Gateway that terminates TLS:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: splunk-s2s
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 9997
      name: tls-s2s
      protocol: TLS
    tls:
      mode: SIMPLE
      credentialName: s2s-tls
    hosts:
    - "*"
  3. Create a Virtual Service for TCP routing:
    CODE 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: splunk-s2s
spec:
  hosts:
  - "*"
  gateways:
  - splunk-s2s
  tcp:
  - match:
    - port: 9997
    route:
    - destination:
        port:
          number: 9997
        host: splunk-standalone-standalone-service

    The Virtual Service no longer handles TLS because the gateway terminates TLS.

  4. Configure your forwarder and indexer or standalone certificates using the documentation: Securing data from forwarders.

Documentation tested on Istio v1.8 and Kubernetes v1.17.

Sticky sessions

Follow the Istio Sticky Sessions documentation to learn how to configure session stickiness for Istio.

Service Mesh and Istio

Istio is a popular choice for its Service Mesh capabilities. However, Service Mesh for Splunk instances is only supported on Istio v1.8 and higher, along with Kubernetes v1.19 and higher.