Custom resources for index and ingestion separation

Custom resource reference for index and ingestion separation.

The Splunk Operator provides Queue, ObjectStorage, IngestorCluster, and IndexerCluster custom resources to configure index and ingestion separation.

Queue

The Queue custom resource stores message queue information shared among IngestorCluster and IndexerCluster resources. The SQS provider is the only supported message queue provider.

Key Type Description
provider string [Required] Provider of message queue (Allowed values: sqs, sqs_cp)
sqs SQS [Required if provider=sqs or provider=sqs_cp] SQS message queue inputs

SQS message queue inputs:

Key Type Description
name string [Required] Name of the queue
authRegion string [Required] Region where the queue is located
endpoint string [Optional, if not provided formed based on authRegion] AWS SQS Service endpoint
dlq string [Required] Name of the dead letter queue
volumes []VolumeSpec [Optional] List of remote storage volumes used to mount the credentials for queue and bucket access (must contain s3_access_key and s3_secret_key)
Note: SOK does not support updates to any of the Queue inputs except the volumes field, which allows changes to secrets.

Example:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Queue
metadata:
  name: queue
spec:
  provider: sqs
  sqs:
    name: sqs-test
    authRegion: us-west-2
    endpoint: https://sqs.us-west-2.amazonaws.com
    dlq: sqs-dlq-test
    volumes:
      - name: s3-sqs-volume
        secretRef: s3-secret

ObjectStorage

The ObjectStorage custom resource stores large messages (messages that exceed the maximum SQS message size) shared among IngestorCluster and IndexerCluster resources. The S3 provider is the only supported object storage provider.

Key Type Description
provider string [Required] Provider of object storage (Allowed values: s3)
s3 S3 [Required if provider=s3] S3 object storage inputs

S3 object storage inputs:

Key Type Description
path string [Required] Remote storage location for messages that are larger than the underlying maximum message size
endpoint string [Optional, if not provided formed based on authRegion] S3-compatible service endpoint
Note: SOK does not support updates to any of the ObjectStorage inputs.

Example:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: ObjectStorage
metadata:
  name: os
spec:
  provider: s3
  s3:
    path: ingestion/smartbus-test
    endpoint: https://s3.us-west-2.amazonaws.com

IngestorCluster

The IngestorCluster custom resource provides high-throughput data ingestion into a durable message queue. Its Splunk pods receive events (outputs.conf) and publish them to a message queue.

In addition to common spec inputs, the IngestorCluster resource provides the following spec configuration parameters:

Key Type Description
replicas integer The number of replicas (defaults to 3)
queueRef corev1.ObjectReference Message queue reference
objectStorageRef corev1.ObjectReference Object storage reference
Note: SOK does not support updates to queueRef and objectStorageRef.
Note: First provisioning or scaling up the number of replicas requires an Ingestor Cluster Splunkd restart. SOK implements this restart automatically.

Example:

The following example configures an IngestorCluster named ingestor with 3 replicas using the SPLUNK_IMAGE_VERSION image. The Queue and ObjectStorage references specify SQS and S3 settings for the ingestion process.

JSON 
apiVersion: enterprise.splunk.com/v4
kind: IngestorCluster
metadata:
  name: ingestor
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  serviceAccount: ingestor-sa 
  replicas: 3
  image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
  queueRef:
    name: queue
  objectStorageRef:
    name: os

IndexerCluster

The IndexerCluster custom resource supports index-only mode, enabling independent scaling, loss-safe buffering, and simplified day-0 and day-n management through Kubernetes CRDs. Its Splunk pods pull events from the queue (inputs.conf) and index them.

In addition to common spec inputs, the IndexerCluster resource provides the following spec configuration parameters:

Key Type Description
replicas integer The number of replicas (defaults to 3)
queueRef corev1.ObjectReference Message queue reference
objectStorageRef corev1.ObjectReference Object storage reference
Note: SOK does not support updates to queueRef and objectStorageRef.
Note: First provisioning or scaling up the number of replicas requires an Indexer Cluster Splunkd restart. SOK implements this restart automatically.

Example:

The following example configures an IndexerCluster named indexer with 3 replicas using the SPLUNK_IMAGE_VERSION image, managed by ClusterManager cm. The Queue and ObjectStorage references specify SQS and S3 settings for the indexing process.

JSON 
apiVersion: enterprise.splunk.com/v4
kind: ClusterManager
metadata:
  name: cm
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  serviceAccount: ingestor-sa 
  image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
---
apiVersion: enterprise.splunk.com/v4
kind: IndexerCluster
metadata:
  name: indexer
  finalizers:
    - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm
  serviceAccount: ingestor-sa
  replicas: 3 
  image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
  queueRef:
    name: queue
  objectStorageRef:
    name: os