Deploy index and ingestion separation

Deploy an index and ingestion separation environment with the Splunk Operator.

Deploy a complete index and ingestion separation environment using the Splunk Operator for Kubernetes, including the Queue, ObjectStorage, IngestorCluster, IndexerCluster, and Horizontal Pod Autoscaler resources.

  • SOK_IMAGE_VERSION: version of the image for Splunk Operator for Kubernetes
  • SPLUNK_IMAGE_VERSION: Splunk Enterprise Docker Image version
  1. Install CRDs and Splunk Operator for Kubernetes.
    CODE 
    make install
    CODE 
    kubectl apply -f ${SOK_IMAGE_VERSION}/splunk-operator-cluster.yaml --server-side

    Verify the operator pod is running:

    CODE 
    kubectl get po -n splunk-operator

    Expected output:

    CODE 
    NAME                                                  READY   STATUS    RESTARTS   AGE
splunk-operator-controller-manager-785b89d45c-dwfkd   2/2     Running   0          4d3h
  2. Create a service account with the required IAM policies.
    CODE 
    eksctl create iamserviceaccount \
  --name ingestor-sa \
  --cluster ind-ing-sep-demo \
  --region us-west-2 \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonSQSFullAccess \
  --approve \
  --override-existing-serviceaccounts

    Verify the service account:

    CODE 
    kubectl describe sa ingestor-sa

    Expected output:

    CODE 
    Name:                ingestor-sa
Namespace:           default
Labels:              app.kubernetes.io/managed-by=eksctl
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/eksctl-ind-ing-sep-demo-addon-iamserviceac-Role1-123456789123
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>
  3. Install the Queue resource.

    Create queue.yaml:

    CODE 
    apiVersion: enterprise.splunk.com/v4
kind: Queue
metadata:
  name: queue
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  provider: sqs
  sqs:
    name: sqs-test
    authRegion: us-west-2
    endpoint: https://sqs.us-west-2.amazonaws.com
    dlq: sqs-dlq-test
    CODE 
    kubectl apply -f queue.yaml

    Verify the Queue resource:

    CODE 
    kubectl get queue

    Expected output:

    CODE 
    NAME   PHASE   AGE   MESSAGE
queue  Ready   20s
  4. Install the ObjectStorage resource.

    Create os.yaml:

    CODE 
    apiVersion: enterprise.splunk.com/v4
kind: ObjectStorage
metadata:
  name: os
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  provider: s3
  s3:
    endpoint: https://s3.us-west-2.amazonaws.com
    path: ingestion/smartbus-test
    CODE 
    kubectl apply -f os.yaml

    Verify the ObjectStorage resource:

    CODE 
    kubectl get os

    Expected output:

    CODE 
    NAME   PHASE   AGE   MESSAGE
os    Ready   20s
  5. Install the IngestorCluster resource.

    Create ingestor.yaml:

    JSON 
    apiVersion: enterprise.splunk.com/v4
kind: IngestorCluster
metadata:
  name: ingestor
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  serviceAccount: ingestor-sa 
  replicas: 3
  image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
  queueRef:
    name: queue
  objectStorageRef:
    name: os
    CODE 
    kubectl apply -f ingestor.yaml

    Verify the pods are running:

    CODE 
    kubectl get po

    Expected output:

    CODE 
    NAME                         READY   STATUS    RESTARTS   AGE
splunk-ingestor-ingestor-0   1/1     Running   0          2m12s
splunk-ingestor-ingestor-1   1/1     Running   0          2m12s
splunk-ingestor-ingestor-2   1/1     Running   0          2m12s

    Verify the ingestor pod configuration by examining the environment variables and configuration files:

    CODE 
    kubectl exec -it splunk-ingestor-ingestor-0 -- sh
sh-4.4$ env | grep AWS
AWS_DEFAULT_REGION=us-west-2
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_REGION=us-west-2
AWS_ROLE_ARN=arn:aws:iam::111111111111:role/eksctl-ind-ing-sep-demo-addon-iamserviceac-Role1-123456789123
AWS_STS_REGIONAL_ENDPOINTS=regional

    Verify the default-mode.conf:

    CODE 
    sh-4.4$ cat /opt/splunk/etc/system/local/default-mode.conf 
[pipeline:remotequeueruleset]
disabled = false

[pipeline:ruleset]
disabled = true

[pipeline:remotequeuetyping]
disabled = false

[pipeline:remotequeueoutput]
disabled = false

[pipeline:typing]
disabled = true

[pipeline:indexerPipe]
disabled = true

    Verify the outputs.conf:

    CODE 
    sh-4.4$ cat /opt/splunk/etc/system/local/outputs.conf 
[remote_queue:sqs-test]
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 4
remote_queue.sqs_smartbus.auth_region = us-west-2
remote_queue.sqs_smartbus.dead_letter_queue.name = sqs-dlq-test
remote_queue.sqs_smartbus.encoding_format = s2s
remote_queue.sqs_smartbus.endpoint = https://sqs.us-west-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-west-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.path = s3://ingestion/smartbus-test
remote_queue.sqs_smartbus.retry_policy = max_count
remote_queue.sqs_smartbus.send_interval = 5s
remote_queue.type = sqs_smartbus
  6. Install the IndexerCluster resource.

    Create idxc.yaml:

    JSON 
    apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
  name: cm
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
  serviceAccount: ingestor-sa 
---
apiVersion: enterprise.splunk.com/v4
kind: IndexerCluster
metadata:
  name: indexer
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
  replicas: 3
  clusterManagerRef:
    name: cm
  serviceAccount: ingestor-sa 
  queueRef:
    name: queue
  objectStorageRef:
    name: os
    CODE 
    kubectl apply -f idxc.yaml

    Verify the pods are running:

    CODE 
    kubectl get po

    Expected output:

    CODE 
    NAME                          READY   STATUS    RESTARTS   AGE
splunk-cm-cluster-manager-0   1/1     Running   0          15m
splunk-indexer-indexer-0      1/1     Running   0          12m
splunk-indexer-indexer-1      1/1     Running   0          12m
splunk-indexer-indexer-2      1/1     Running   0          12m
splunk-ingestor-ingestor-0    1/1     Running   0          27m
splunk-ingestor-ingestor-1    1/1     Running   0          29m
splunk-ingestor-ingestor-2    1/1     Running   0          31m

    Verify the indexer pod configuration by examining the inputs.conf:

    CODE 
    kubectl exec -it splunk-indexer-indexer-0 -- sh
sh-4.4$ cat /opt/splunk/etc/system/local/inputs.conf 

[splunktcp://9997]
disabled = 0

[remote_queue:sqs-test]
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 4
remote_queue.sqs_smartbus.auth_region = us-west-2
remote_queue.sqs_smartbus.dead_letter_queue.name = sqs-dlq-test
remote_queue.sqs_smartbus.endpoint = https://sqs.us-west-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-west-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.path = s3://ingestion/smartbus-test
remote_queue.sqs_smartbus.retry_policy = max_count
remote_queue.type = sqs_smartbus

    Verify the default-mode.conf:

    CODE 
    sh-4.4$ cat /opt/splunk/etc/system/local/default-mode.conf 
[pipeline:remotequeueruleset]
disabled = false

[pipeline:ruleset]
disabled = true

[pipeline:remotequeuetyping]
disabled = false

[pipeline:remotequeueoutput]
disabled = false

[pipeline:typing]
disabled = true
  7. Install the Horizontal Pod Autoscaler for the IngestorCluster.

    Create hpa-ing.yaml:

    CODE 
    apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ing-hpa
spec:
  scaleTargetRef:
    apiVersion: enterprise.splunk.com/v4
    kind: IngestorCluster
    name: ingestor
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
    CODE 
    kubectl apply -f hpa-ing.yaml

    Verify the HPA:

    CODE 
    kubectl get hpa

    Expected output:

    CODE 
    NAME      REFERENCE                  TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
ing-hpa   IngestorCluster/ingestor   cpu: <unknown>/50%   3         10        0          10s
  8. Generate fake load to verify autoscaling.

    Retrieve the HEC token:

    CODE 
    kubectl get secret splunk-default-secret -o yaml

    Decode the token:

    CODE 
    echo HEC_TOKEN | base64 -d

    Create loadgen.yaml with a ConfigMap containing a Locust load test script and a CronJob to run it:

    PYTHON 
    apiVersion: v1
kind: ConfigMap
metadata:
  name: hec-locust-config
data:
  requirements.txt: |
    locust
    requests
    urllib3

  locustfile.py: |
    import urllib3
    from locust import HttpUser, task, between

    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

    class HECUser(HttpUser):
        wait_time = between(1, 2)
        host = "https://splunk-ingestor-ingestor-service:8088"

        def on_start(self):
            self.client.verify = False

        @task
        def send_event(self):
            token = "HEC_TOKEN"
            headers = {
                "Authorization": f"Splunk {token}",
                "Content-Type": "application/json"
            }
            payload = {"event": {"message": "load test", "value": 123}}
            self.client.post(
                "/services/collector/event",
                json=payload,
                headers=headers,
                name="HEC POST"
            )
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: hec-locust-load
spec:
  schedule: "*/2 * * * *"
  concurrencyPolicy: Replace
  startingDeadlineSeconds: 60
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          containers:
          - name: locust
            image: python:3.9-slim
            command:
              - sh
              - -c
              - |
                pip install --no-cache-dir -r /app/requirements.txt \
                  && exec locust \
                     -f /app/locustfile.py \
                     --headless \
                     -u 200 \
                     -r 50 \
                     --run-time 1m50s
            volumeMounts:
            - name: app
              mountPath: /app
          restartPolicy: OnFailure
          volumes:
          - name: app
            configMap:
              name: hec-locust-config
              defaultMode: 0755
    CODE 
    kubectl apply -f loadgen.yaml

    Monitor the autoscaling behavior. Under load, the HPA scales the IngestorCluster replicas up:

    CODE 
    kubectl get hpa
NAME      REFERENCE                  TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
ing-hpa   IngestorCluster/ingestor   cpu: 115%/50%   3         10        10         8m54s

    Verify that the large message store on S3 receives data:

    CODE 
    aws s3 ls s3://ingestion/smartbus-test/

You have deployed a complete index and ingestion separation environment with the Splunk Operator. The IngestorCluster receives events and publishes them to the SQS queue, while the IndexerCluster pulls events from the queue and indexes them. The Horizontal Pod Autoscaler scales the IngestorCluster based on CPU utilization.