Set up the REMOTE cluster for federated search

Configure the REMOTE search head cluster as the data source for federated search.

The REMOTE cluster is the federated search data source. Configure it with a role, a service account, and an exposed management endpoint that the LOCAL cluster can reach.

  1. Select a REMOTE search head cluster pod and credentials.

    Don't assume search-head-0 holds captaincy. Captaincy changes dynamically over time, so select a healthy pod at runtime instead of hardcoding an ordinal.

    CODE 
    NAMESPACE="stos-auto"
REMOTE_POD=$(kubectl -n $NAMESPACE get pods \
  -l app.kubernetes.io/instance=splunk-remote-shc-search-head \
  -o jsonpath='{.items[0].metadata.name}')
REMOTE_ADMIN=$(kubectl -n $NAMESPACE get secret splunk-remote-shc-search-head-secret-v1 \
  -o jsonpath='{.data.password}' | base64 -d)

    Replace stos-auto and remote-shc with the namespace and custom resource name from your environment before you run the remaining commands.

  2. Create a federated search role with access to the indexes you want to share.
    CODE 
    # Create fsh_user role with access to every remote index you want to expose
kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "admin:$REMOTE_ADMIN" \
  -X POST "https://localhost:8089/services/authorization/roles/fsh_user" \
  -d "srchIndexesAllowed=_audit,demo" \
  -d "srchIndexesDefault=_audit" \
  -d "imported_roles=user"

    Specify srchIndexesAllowed as the full comma-separated list of remote indexes the federated user can access. Specify srchIndexesDefault as the default index for searches. When you update srchIndexesAllowed later, include the complete desired list again.

  3. Create a dedicated service account for federated authentication.
    CODE 
    # Create fsh_svc service account with fsh_user role
kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- \
  /opt/splunk/bin/splunk add user fsh_svc \
  -password 'SvcP@ssw0rd' \
  -role fsh_user \
  -auth "admin:$REMOTE_ADMIN"
    Important: Use a strong, unique password and store credentials in a Kubernetes Secret rather than plain text. Rotate credentials regularly and grant only the permissions the service account requires.
  4. Create a Service resource of type ExternalName to expose the REMOTE management endpoint.
    CODE 
    # remote-mgmt-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: remote-mgmt
  namespace: stos-auto
spec:
  type: ExternalName
  externalName: ingress-nginx-controller.ingress-nginx.svc.cluster.local
  ports:
  - port: 443
    targetPort: 443
    protocol: TCP

    Apply the service:

    CODE 
    kubectl apply -f remote-mgmt-service.yaml
  5. Create an Ingress resource to route traffic to the REMOTE search head cluster.
    CODE 
    # remote-mgmt-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: remote-mgmt-ingress
  namespace: stos-auto
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: remote-mgmt.stos-auto.svc.cluster.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: splunk-remote-shc-search-head-service
            port:
              number: 8089

    Apply the ingress:

    CODE 
    kubectl apply -f remote-mgmt-ingress.yaml
  6. Verify the REMOTE setup.
    CODE 
    # Test service account authentication
kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "fsh_svc:SvcP@ssw0rd" \
  "https://localhost:8089/services/server/info?output_mode=json" | \
  grep -o '"federated_search_enabled":[^,]*'

# Test search capability
kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "fsh_svc:SvcP@ssw0rd" \
  -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \
  -d "search=search index=_audit | head 1" \
  -d "exec_mode=oneshot"
    The first command returns "federated_search_enabled":true. The second command returns search results without permission errors.