Troubleshooting issues with physical separation

Identify and isolate issues in the Kubernetes, SmartBus, and Splunk Enterprise layers to troubleshoot problems with physical separation.

When troubleshooting physical separation, isolate the failing layer before changing the configuration. Isolating the issue helps identify whether the issue is related to Kubernetes resource management, cloud-service access, or Splunk data processing. Check the layers in the following order:

Splunk Operator for Kubernetes (SOK) and Kubernetes layer

Check the status and events for the custom resources (CRs), including the Queue, ObjectStorage, and IngestorCluster resources. Review the operator controller logs for reconciliation errors.

Also verify the following:

  • The resources report the expected status and readiness.

  • The IngestorCluster pods are scheduled on available worker nodes.

  • The pods start successfully and remain running.

  • The operator completes the expected resource lifecycle and reconciliation operations.

SmartBus layer

If the Queue, ObjectStorage, and IngestorClusterresources are ready, but data is not moving between the ingestion and indexing tiers, check the SmartBus dependencies:

  • Verify the Amazon Simple Queue Service (SQS) endpoint.

  • Verify that the ingestion and indexing tiers have the required queue permissions.

  • Check the dead-letter queue (DLQ) for messages that could not be processed.

  • Verify the ingestion object-store path and permissions.

  • Check queue depth and processing lag.

Splunk Enterprise layer

If the queue and object store are operating normally, check the Splunk data flow:

  • Verify that HTTP Event Collector (HEC) and Splunk-to-Splunk (S2S) inputs route data to the IngestorCluster.

  • Review the generated Splunk Enterprise configuration for incorrect or missing settings.

  • Verify that the ingestion tier publishes data and that the indexing tier consumes it.

  • Compare ingestion throughput with queue-consumption throughput.

  • Compare the number of events received with the number of events indexed and searchable.