Configuration best practices

Review configuration best practices for logical and physical separation, including multisite storage, encryption, independent scaling, SOK lifecycle management, and forwarder acknowledgments.

Keep the following important points in mind when you deploy Ingest-Tier Scaling:

Logical separation

Keep the following best practices in mind when configuring logical separation:

  • For multisite indexer clusters, use separate SQS queues and S3 buckets for smartbus blobs per site.
    • This ensures that the indexers from each site share the ingestion load with fellow indexers from the same site and do not compete with remote site indexers.
    • This may add ingestion latency for indexers from the far site.
    • This also means that the SQS and smartbus S3 bucket settings are asymmetric for different sites; therefore, you may need to push them through REST APIs to each indexer (instead of a cluster bundle push) as a one-time provisioning.
  • Use the S2S encoding format for blobs. Tests have shown S2S to be faster than Protobuf in the Splunk Cloud Platform service.
  • We highly recommend using the SSE-C encryption scheme to implement better security measures and protect the ingest blobs.
  • We highly recommend that forwarders and HEC inputs use useACK. The default send_interval is 4 seconds, and ingestors can crash during this interval, which can lead to data loss. With useACK, the system no longer creates back pressure, and the performance penalty is minimal because the system sends the ACK as soon as the message arrives on the smart bus.
  • For single-site indexer clusters, you can push smartbus configuration through a cluster bundle push.

Physical separation

Keep the following best practices in mind when configuring physical separation:

  • Use a separate queue and ingestion object-store location for each site in multisite deployments.

  • Use the encryption method supported by your cloud provider and deployment.

  • Treat the ingestion object store as separate from the object store used by SmartStore for warm buckets.

  • Scale the IngestorCluster and the indexer cluster according to their individual workloads. Scaling the IngestorCluster does not automatically scale the indexer cluster.

  • Use supported Splunk Operator for Kubernetes (SOK) workflows to deploy, monitor, scale, and restart the IngestorCluster. See Configure physical separation of indexing and ingestion with SOK.

  • If forwarders send Splunk-to-Splunk (S2S) data to the IngestorCluster, configure useACK=true in the [tcpout:<target_group>] stanza of outputs.conf when forwarder indexer acknowledgment is supported and required for your deployment. Forwarder acknowledgment confirms handling by the receiving Splunk endpoint; it does not by itself confirm that the data has completed asynchronous SmartBus processing and indexing.