Clustered deployment examples

Deployment examples for Splunk Enterprise clusters on Kubernetes.

Examples for creating and managing clustered Splunk Enterprise deployments using the Splunk Operator, including indexer clusters, search head clusters, and monitoring console.

Creating a Standalone instance

The two basic building blocks of Splunk Enterprise infrastructure are search heads and indexers. Use a Standalone resource to create a single instance that can perform either, or both of these roles. The following example creates a deployment named single, used in the search head and cleanup examples later in this topic. For a complete walkthrough of creating, verifying, and cleaning up a Standalone deployment, see Create a Standalone instance.

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: single
  finalizers:
  - enterprise.splunk.com/delete-pvc
Note: The Splunk Operator generates instance passwords automatically. To review the passwords, refer to the documentation for reading the global Kubernetes secret object.

Indexer Clusters

When customers outgrow the capabilities of a single instance for indexing and search, they scale the infrastructure up to an indexer cluster. The Splunk Operator makes creation of a cluster easy by using a ClusterManager resource for Cluster Manager, and using the IndexerCluster resource for the cluster peers:

Cluster Manager

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
  name: cm
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  monitoringConsoleRef:
    name: example-mc
EOF

The Splunk Operator is responsible for configuring and maintaining the connection between the cluster manager and the index cluster peers, but it does not manage Splunk Apps. The cluster manager manages the Splunk Apps and Add-ons distributed to all peers in the indexer cluster. See the documentation for installing Splunk Apps for more information.

The Splunk Operator also controls the upgrade cycle, and implements the recommended order of cluster manager, search heads, and indexers, by defining and updating the Docker image used by each IndexerCluster part.

This example includes the monitoringConsoleRef parameter used to define a monitoring console pod. The monitoring console pod does not need to be running; the name can be predefined and the pod started later. To start the monitoring console pod, refer to the Monitoring Console section, or use the following example:

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: MonitoringConsole
metadata:
  name: example-mc
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
EOF

Building a multisite cluster follows a similar process: define a different zone affinity and site in each child IndexerCluster resource. See multisite cluster examples in the Splunk Enterprise documentation.

Note: The Splunk Operator generates instance passwords automatically. To review the passwords, refer to the documentation for reading the global Kubernetes secret object.

Indexer cluster peers

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: IndexerCluster
metadata:
  name: example
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm
  monitoringConsoleRef:
    name: example-mc
EOF

This automatically configures a cluster, with a predetermined number of index cluster peers generated automatically based upon the replication_factor (RF) set. This example includes the monitoringConsoleRef parameter used to define a monitoring console pod.

Note: If you try to specify the number of replicas on an IndexerCluster CR less than the RF (as set on ClusterManager,) the Splunk Operator always scales the number of peers to either the replication_factor for single site indexer clusters, or to the origin count in site_replication_factor for multi-site indexer clusters. The minimum number of indexer cluster peers is 3.

After starting the cluster manager, indexer cluster peers, and monitoring console pods using the examples above, use the kubectl get pods command to verify your environment:

CODE 
$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS    AGE
splunk-cm-cluster-manager-0                  1/1     Running   0          29s
splunk-example-indexer-0                    1/1     Running   0          29s
splunk-example-indexer-1                    1/1     Running   0          29s
splunk-example-indexer-2                    1/1     Running   0          29s
splunk-example-mc-monitoring-console-0      1/1     Running   0          40s
splunk-operator-7c5599546c-wt4xl            1/1     Running   0          14h

Scaling cluster peers using replicas

If you want to add more indexers as cluster peers, update your IndexerCluster CR and define the replicas parameter:

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: IndexerCluster
metadata:
  name: example
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm
  replicas: 3
  monitoringConsoleRef:
    name: example-mc
EOF
CODE 
$ kubectl get pods
NAME                                         READY    STATUS    RESTARTS   AGE
splunk-cm-cluster-manager-0                    1/1     Running   0          14m
splunk-example-indexer-0                      1/1     Running   0          14m
splunk-example-indexer-1                      1/1     Running   0          70s
splunk-example-indexer-2                      1/1     Running   0          70s
splunk-example-mc-monitoring-console-0        1/1     Running   0          80s
splunk-operator-7c5599546c-wt4xl              1/1     Running   0          14h

You can now easily scale your indexer cluster by patching the replicas count. For example:

JSON 
$ kubectl patch indexercluster example --type=json -p '[{"op": "replace", "path": "/spec/replicas", "value": 5}]'
indexercluster.enterprise.splunk.com/example patched

For efficiency, note that you can use the following short names with kubectl:

  • clustermanager: cmanager-idxc
  • indexercluster: idc or idxc
  • searchheadcluster: shc
  • LicenseManager: lm
  • monitoringconsole: mc

All CRs that support a replicas field can be scaled using the kubectl scale command. For example:

CODE 
$ kubectl scale idc example --replicas=5
indexercluster.enterprise.splunk.com/example scaled

Scaling cluster peers using pod autoscaling

You can also create Horizontal Pod Autoscalers to manage dynamic scaling for you. For example:

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: idc-example
  namespace: splunk-operator
spec:
  scaleTargetRef:
    apiVersion: enterprise.splunk.com/v3
    kind: IndexerCluster
    name: example
  minReplicas: 5
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50
EOF
CODE 
$ kubectl get hpa
NAME          REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
idc-example   IndexerCluster/example   16%/50%   5         10        5          15m

Specifying pod disruption budgets (PDB) for Splunk Enterprise Deployments

You can also create Pod Disruption Budgets to help you run highly available applications even during disruptions. For dynamic replica scaling, see Horizontal Pod Autoscaling. For example you can target a PDB at a standalone which ensures at least three standalone pods are present despite disruptions:

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: demo
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 3
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: stdaln-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: splunk-demo-standalone
EOF
CODE 
$ kubectl describe pdb stdaln-pdb
Name:           stdaln-pdb
Namespace:      splunk-operator
Min available:  3
Selector:       app.kubernetes.io/instance=splunk-demo-standalone
Status:
    Allowed disruptions:  0
    Current:              3
    Desired:              3
    Total:                3
Events:                   <none>
Note: This targets the standalone using the labels that the Splunk Operator uniquely creates for the Standalone CR instance. You can use a similar procedure for other Splunk Enterprise CRs.

Create a search head for your index cluster

To create a standalone search head that is configured to search your indexer cluster, add the clusterManagerRef parameter:

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: single
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm
  monitoringConsoleRef:
    name: example-mc
EOF

Note that the clusterManagerRef field points to the cluster manager for the indexer cluster. This example includes the monitoringConsoleRef parameter used to define a monitoring console pod.

CODE 
$ kubectl get pods
NAME                                         READY    STATUS    RESTARTS   AGE
splunk-cm-cluster-manager-0                    1/1     Running   0          14m
splunk-example-indexer-0                      1/1     Running   0          14m
splunk-example-indexer-1                      1/1     Running   0          70s
splunk-example-indexer-2                      1/1     Running   0          70s
splunk-example-mc-monitoring-console-0        1/1     Running   0          80s
splunk-single-standalone-0                    1/1     Running   0          90s
splunk-operator-7c5599546c-wt4xl              1/1     Running   0          14h

Another Cluster Manager example

Having a separate CR for cluster manager allows you to define parameters differently than the indexers, such as storage capacity and the storage class used by persistent volumes.

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
  name: cm
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  etcVolumeStorageConfig:
    storageClassName: gp2
    storageCapacity: 15Gi
  varVolumeStorageConfig:
    storageClassName: customStorageClass
    storageCapacity: 25Gi
---
apiVersion: enterprise.splunk.com/v3
kind: IndexerCluster
metadata:
  name: idxc-part1
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  # No cluster-manager created, uses the referenced one
  clusterManagerRef:
    name: cm
  replicas: 3
  storageClassName: local
  varStorage: "128Gi"
  monitoringConsoleRef:
    name: example-mc
EOF

Monitoring Console

The Monitoring Console provides detailed topology and performance information about your Splunk Enterprise deployment. Reference the monitoring console (MC) pod by using the monitoringConsoleRef parameter. When you create or delete a pod that references the monitoringConsoleRef parameter, the MC pod automatically updates itself and creates or removes connections to those pods.

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: MonitoringConsole
metadata:
  name: example-mc
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
EOF

There is no preferred order when running an MC pod; you can start the pod before or after the other CRs in the namespace. To associate a new MC pod with an existing CR that does not define the monitoringConsoleRef, you can patch those CRs and add it. For example:

JSON 
kubectl patch cm-idxc cm --type=json -p '[{"op":"add", "path":"/spec/monitoringConsoleRef/name", "value":"example-mc"}]'

for a cluster manager and

JSON 
kubectl patch shc test --type=json -p '[{"op":"add", "path":"/spec/monitoringConsoleRef/name", "value":"example-mc"}]'

for a search head cluster.

Search Head Clusters

A search head cluster is used to distribute users and search load across multiple instances, and provides high availability for search jobs. See About search head clustering in the Splunk Enterprise documentation.

You can create a search head cluster that is configured to communicate with your indexer cluster by using a SearchHeadCluster resource and adding the clusterManagerRef parameter.

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: SearchHeadCluster
metadata:
  name: example
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm
  monitoringConsoleRef:
    name: example-mc
EOF

This automatically creates a deployer with 3 search heads clustered together. Search head clusters require a minimum of 3 members. This example includes the monitoringConsoleRef parameter and name used to define a monitoring console (MC) pod. To start the monitoring console pod, refer to the Monitoring Console section, or use the following example:

CODE 
cat <<EOF | kubectl apply -f -
apiVersion: enterprise.splunk.com/v4
kind: MonitoringConsole
metadata:
  name: example-mc
  namespace: splunk-operator
  finalizers:
  - enterprise.splunk.com/delete-pvc
EOF
CODE 
$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
splunk-cm-cluster-manager-0                   1/1     Running   0          53m
splunk-example-deployer-0                    0/1     Running   0          29s
splunk-example-indexer-0                     1/1     Running   0          53m
splunk-example-indexer-1                     1/1     Running   0          40m
splunk-example-indexer-2                     1/1     Running   0          40m
splunk-example-indexer-3                     1/1     Running   0          37m
splunk-example-indexer-4                     1/1     Running   0          37m
splunk-example-search-head-0                 0/1     Running   0          29s
splunk-example-search-head-1                 0/1     Running   0          29s
splunk-example-search-head-2                 0/1     Running   0          29s
splunk-operator-7c5599546c-pmbc2             1/1     Running   0          12m
splunk-single-standalone-0                   1/1     Running   0          11m
splunk-example-mc-monitoring-console-0       1/1     Running   0          80s

Similar to indexer clusters, you can scale a search head cluster by patching the replicas parameter.

Note: The Splunk Operator generates instance passwords automatically. To review the passwords, refer to the documentation for reading the global Kubernetes secret object.

Cluster services

The creation of SearchHeadCluster, ClusterManager, MonitoringConsole, and IndexerCluster resources also creates corresponding Kubernetes services:

CODE 
$ kubectl get svc
NAME                                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                          AGE
splunk-cm-cluster-manager-service                            ClusterIP   10.100.98.17     <none>        8000/TCP,8089/TCP                                55m
splunk-cm-indexer-service                                   ClusterIP   10.100.119.27    <none>        8000/TCP,8089/TCP                                55m
splunk-example-mc-monitoring-console-service                ClusterIP   10.100.7.28      <none>        8000/TCP,8088/TCP,8089/TCP,9997/TCP              54m
splunk-example-deployer-service                             ClusterIP   10.100.43.240    <none>        8000/TCP,8089/TCP                                118s
splunk-example-indexer-headless                             ClusterIP   None             <none>        8000/TCP,8088/TCP,8089/TCP,9997/TCP              55m
splunk-example-indexer-service                              ClusterIP   10.100.192.73    <none>        8000/TCP,8088/TCP,8089/TCP,9997/TCP              55m
splunk-example-search-head-headless                         ClusterIP   None             <none>        8000/TCP,8089/TCP                                118s
splunk-example-search-head-service                          ClusterIP   10.100.37.53     <none>        8000/TCP,8089/TCP                                118s
splunk-operator-metrics                                     ClusterIP   10.100.181.146   <none>        8383/TCP,8686/TCP                                11d

To log in to your new Splunk Enterprise cluster, you can forward port 8000 to one of the search head pods, or use a load balancing service that is automatically created for your deployment:

CODE 
kubectl port-forward service/splunk-example-search-head-service 8000

Similar to other examples, you can obtain the default administrator password from the global Kubernetes secrets object:

CODE 
kubectl get secret splunk-<namespace>-secret -o jsonpath='{.data.password}' | base64 --decode

See the documentation for configuring Ingress for guidance on making your Splunk Enterprise clusters accessible from outside of Kubernetes.

Clean up resources

As these examples show, the Splunk Operator makes it easy to create and manage clustered deployments of Splunk Enterprise. Given the reduced complexity, the comparable resource requirements from using containers, and the ability to easily start small and scale as necessary, use the IndexerCluster and SearchHeadCluster resources when creating deployments using the Splunk Operator.

To remove the resources created from this example, run:

CODE 
kubectl delete -n splunk-operator standalone single
kubectl delete -n splunk-operator shc example
kubectl delete -n splunk-operator idc example
kubectl delete -n splunk-operator mc example-mc
kubectl delete -n splunk-operator clustermanager cm
Note: When you plan new deployments, prefer the IndexerCluster and SearchHeadCluster resources over ad hoc combinations of standalones.