Configuring Ingest-Tier Scaling

To configure Ingest-Tier Scaling, use the following procedures.

Configure smartbus queue

  1. Go to Amazon SQS and click on create queue.
  2. Give the queue a name and adjust the visibility parameters, message retention period, and so on.
  3. Choose the encryption scheme and access policy. Splunk Cloud Platform uses SSE-C for enhanced security and recommends this approach for customer-managed deployments.
  4. Configure the Dead Letter Queue (DLQ) to another SQS queue that you may have created to deliver failed messages to that queue after a number of failed attempts to receive.
  5. Click the “Create queue” button.
  6. For indexer clusters with multiple sites, create a separate SQS queue for each site.
  7. The SQS queues hold messages that store only the metadata for accessing the associated actual ingestion blob, which resides in the SmartStore.

Configure SmartStore bucket

Important: Do Not Apply to SmartStore Warm Buckets. The S3 bucket configured in this section is used only for ingest-tier scaling. Do not create separate S3 buckets for SmartStore warm buckets based on this guidance. SmartStore warm bucket configuration is unchanged and follows existing SmartStore deployment best practices.
  1. This process is similar to creating a bucket in S3 or Azure when we enable the SmartStore for Splunk warm buckets.
  2. Create a separate S3 bucket to store the ingestion blobs (we refer to this path as the large message store in the settings for Ingest-Tier Scaling).
  3. For indexer clusters with multiple sites, create a separate S3 bucket for each site.

Configure Key Management System key

  1. Although the large message store bucket supports different encryption schemes, we recommend using SSE-C. SSE-C employs server-side encryption with a client key and provides greater security. See SmartStore on S3 security strategies.
  2. The Key Management System (KMS) key provides access to the SmartStore bucket.
  3. Because the large message store stores the actual customer data in the form of ingestion blobs, we must enable the highest form of security available (which is SSE-C).
  4. For this, configure the KMS key and provide it in the large message store settings for Ingest-Tier Scaling.
  5. Configuring a KMS key is the same process as enabling the SmartStore for Splunk warm buckets in S3.

Enable Ingest-Tier Scaling

By default, Ingest-Tier Scaling is not enabled. After you complete the preceding configuration steps, you can then enable Splunk with Ingest-Tier Scaling.