Logical separation of indexing and ingestion

You can enable Ingest-Tier Scaling input and output configurations in the same indexer instance. This approach logically decouples indexing and ingestion. Benefits of this approach include the following:

  1. Can help to enhance data safety (with useACK).
  2. Causes less queue blockage in the forwarding plane (up to the typing queue).

Enable the pipeline processors for remote queue default-mode.conf


[pipeline:remotequeueruleset]
disabled = false
[pipeline:ruleset]
disabled = true
[pipeline:remotequeuetyping]
disabled = false
[pipeline:remotequeueoutput]
disabled = false
[pipeline:typing]
disabled = true

Enable the remote output queue outputs.conf


[remote_queue:<queue-name>]
remote_queue.type = sqs_smartbus
remote_queue.sqs_smartbus.encoding_format = s2s
remote_queue.sqs_smartbus.auth_region = us-east-2
remote_queue.sqs_smartbus.access_key = <REDACTED>
remote_queue.sqs_smartbus.secret_key = <REDACTED>
remote_queue.sqs_smartbus.endpoint = https://sqs.us-east-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-east-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.path = s3://user/smartbus-test
remote_queue.sqs_smartbus.large_message_store.encryption_scheme = sse-c
remote_queue.sqs_smartbus.large_message_store.key_id = <REDACTED>
remote_queue.sqs_smartbus.large_message_store.key_refresh_interval = 24h
remote_queue.sqs_smartbus.large_message_store.kms_endpoint = https://kms.us-east-2.amazonaws.com
remote_queue.sqs_smartbus.dead_letter_queue.name = <dlq-queue-name>
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 3
remote_queue.sqs_smartbus.retry_policy = max_count

Enable the remote input queue inputs.conf


[remote_queue:<queue-name>]
remote_queue.type = sqs_smartbus
remote_queue.sqs_smartbus.auth_region = us-east-2
remote_queue.sqs_smartbus.access_key = <REDACTED>
remote_queue.sqs_smartbus.secret_key = <REDACTED>
remote_queue.sqs_smartbus.endpoint = https://sqs.us-east-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-east-2.amazonaws.com
remote_queue.sqs_smartbus.large_message_store.path = s3://user/smartbus-test
remote_queue.sqs_smartbus.large_message_store.encryption_scheme = sse-c
remote_queue.sqs_smartbus.large_message_store.key_id = <REDACTED>
remote_queue.sqs_smartbus.large_message_store.key_refresh_interval = 24h
remote_queue.sqs_smartbus.large_message_store.kms_endpoint = https://kms.us-east-2.amazonaws.com
remote_queue.sqs_smartbus.dead_letter_queue.name = <dlq-queue-name>
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 3

Indexer cluster bundle push single site / multisite

If the indexer cluster is a single site, you can use a normal indexer cluster bundle push to push the settings for Ingest-Tier Scaling. The bundle push triggers a rolling restart for each indexer.

If the indexer cluster is a multisite, you must manually configure the settings on each indexer.

Verifying that Ingest-Tier Scaling is enabled

After you push the Ingest-Tier Scaling settings and the instances finish restarting, examine the metrics to determine if events route through the smart bus.


index=_internal source=*/metrics.log group=smartbus name=ingestor series=upload upload_type=datastore | timechart span=5m sum(successful_message_count) as uploadStore | appendcols [|search index=_internal source=*/metrics.log group=smartbus name=ingestor series=upload upload_type=bus| timechart span=5m sum(successful_message_count) as uploadBus] | appendcols [|search index=_internal source=*/metrics.log group=smartbus name=indexer series=download download_type=datastore | timechart span=5m sum(successful_message_count) as downloadStore] | appendcols [|search index=_internal source=*/metrics.log group=smartbus name=indexer series=download download_type=bus | timechart span=5m sum(successful_message_count) as downloadBus]