Test and validate federated search

Test federated search from the LOCAL cluster and confirm it returns REMOTE data.

1. Verify that the LOCAL cluster can reach the REMOTE management API.
   CODECopy

   kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- curl -sk \ -u "fsh_svc:SvcP@ssw0rd" \ "https://remote-mgmt.stos-auto.svc.cluster.local:443/services/server/info?output_mode=json" | \ grep -o '"federated_search_enabled":[^,]*'

   kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- curl -sk \
     -u "fsh_svc:SvcP@ssw0rd" \
     "https://remote-mgmt.stos-auto.svc.cluster.local:443/services/server/info?output_mode=json" | \
     grep -o '"federated_search_enabled":[^,]*'

   The command returns "federated_search_enabled":true.
2. Verify that data exists on the REMOTE cluster.
   CODECopy

   kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \ -u "fsh_svc:SvcP@ssw0rd" \ -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \ -d "search=search index=_audit | stats count" \ -d "exec_mode=oneshot"

   kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
     -u "fsh_svc:SvcP@ssw0rd" \
     -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \
     -d "search=search index=_audit | stats count" \
     -d "exec_mode=oneshot"

   The command returns a count greater than 0.
3. Verify that a federated search from the LOCAL cluster returns REMOTE data.
   CODECopy

   kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- curl -sk \ -u "admin:$LOCAL_ADMIN" \ -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \ -d "search=search index=federated:r_audit | head 10" \ -d "exec_mode=oneshot"

   kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- curl -sk \
     -u "admin:$LOCAL_ADMIN" \
     -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \
     -d "search=search index=federated:r_audit | head 10" \
     -d "exec_mode=oneshot"

   The results contain data from the REMOTE cluster:

   • The splunk_federated_provider field is set to remote_thru_nginx.
   • The splunk_server field shows the REMOTE indexer names.
4. Verify that statistics work across a federated index.
   CODECopy

   kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- curl -sk \ -u "admin:$LOCAL_ADMIN" \ -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \ -d "search=search index=federated:r_audit | stats count by sourcetype" \ -d "exec_mode=oneshot"

   kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- curl -sk \
     -u "admin:$LOCAL_ADMIN" \
     -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \
     -d "search=search index=federated:r_audit | stats count by sourcetype" \
     -d "exec_mode=oneshot"

   The command returns event counts grouped by sourcetype.
5. Verify federated search in Splunk Web.
   JSONCopy

   kubectl -n $NAMESPACE port-forward svc/splunk-local-shc-search-head-service 8000:8000 kubectl -n $NAMESPACE get secret splunk-local-shc-search-head-secret-v1 \ -o jsonpath='{.data.password}' | base64 -d

   kubectl -n $NAMESPACE port-forward svc/splunk-local-shc-search-head-service 8000:8000
   kubectl -n $NAMESPACE get secret splunk-local-shc-search-head-secret-v1 \
     -o jsonpath='{.data.password}' | base64 -d

   Open http://localhost:8000 in a browser, log in with the admin credentials, and run the search index=federated:r_audit | stats count.