Troubleshoot insufficient permission errors

Resolve permission errors returned by federated search queries.

Federated search queries return the error insufficient permission to access this resource.

The federated service account role or the local user role is missing required search or index permissions. The built-in admin role already includes the search capability, so avoid editing the default admin role just to enable federated search.

  1. Check the remote service account role.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "admin:$REMOTE_ADMIN" \
  "https://localhost:8089/services/authorization/roles/fsh_user?output_mode=json" | \
  grep -E '"imported_roles"|"srchIndexesAllowed"'
  2. If needed, update remote index permissions by sending the full allowed-index list.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "admin:$REMOTE_ADMIN" \
  -X POST "https://localhost:8089/services/authorization/roles/fsh_user" \
  -d "srchIndexesAllowed=_audit,demo" \
  -d "srchIndexesDefault=_audit" \
  -d "imported_roles=user"

Troubleshoot federated provider not found

Resolve "Not Found" errors returned when the federated search app isn't deployed on the LOCAL cluster.

Federated search requests return {"messages":[{"type":"ERROR","text":"Not Found"}]}.

The federated search app isn't deployed on the LOCAL cluster, or Splunk hasn't restarted since deployment.

  1. Check whether the app exists.
    CODE 
    kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- \
  ls /opt/splunk/etc/apps/federated-search-app
  2. Restart Splunk.
    CODE 
    kubectl -n $NAMESPACE delete pod -l app.kubernetes.io/instance=splunk-local-shc-search-head

Troubleshoot cannot reach remote SHC

Resolve connection timeouts between the LOCAL and REMOTE search head clusters.

The LOCAL cluster can't reach the REMOTE search head cluster. Symptoms include connection timeouts or a Cannot reach remote SHC error.

The Service resource of type ExternalName or the Ingress resource is misconfigured.

  1. Verify that the Service resource exists.
    CODE 
    kubectl -n $NAMESPACE get svc remote-mgmt
  2. Verify that the Ingress resource exists.
    CODE 
    kubectl -n $NAMESPACE get ingress
  3. Test connectivity from the LOCAL pod.
    CODE 
    kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- \
  curl -sk "https://remote-mgmt.stos-auto.svc.cluster.local:443"

Troubleshoot authentication failures

Resolve 401 Unauthorized and authentication failure errors for the federated search service account.

Federated search requests return 401 Unauthorized or Authentication failed.

The service account credentials are incorrect.

  1. Verify that the service account exists on the REMOTE cluster.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- \
  /opt/splunk/bin/splunk list user -auth admin:<password> | grep fsh_svc
  2. Test authentication directly.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "fsh_svc:SvcP@ssw0rd" \
  "https://localhost:8089/services/authentication/current-context?output_mode=json"

Troubleshoot no data returned

Resolve a federated search that returns 0 results.

A federated search returns 0 results.

The index doesn't exist on the REMOTE cluster, the remote index has no data, or the fsh_user role doesn't have permission to the index.

  1. Check whether the index exists on the REMOTE cluster.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- \
  /opt/splunk/bin/splunk list index -auth admin:<password> | grep _audit
  2. Verify the fsh_user role permissions.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "admin:<password>" \
  "https://localhost:8089/services/authorization/roles/fsh_user?output_mode=json" | \
  grep srchIndexesAllowed
  3. Test a direct search on the REMOTE cluster as fsh_svc.
    CODE 
    kubectl -n $NAMESPACE exec $REMOTE_POD -c splunk -- curl -sk \
  -u "fsh_svc:SvcP@ssw0rd" \
  -X POST "https://localhost:8089/services/search/jobs?output_mode=json" \
  -d "search=search index=_audit | stats count" \
  -d "exec_mode=oneshot"

Troubleshoot app framework not deploying apps

Resolve App Framework failing to deploy the federated search app to the LOCAL cluster after uploading it to Azure.

App Framework doesn't deploy the federated search app to the LOCAL cluster after you upload it to Azure.

Azure permissions or the App Framework path configuration is incorrect.

  1. Check the operator logs.
    CODE 
    kubectl -n splunk-operator logs -l control-plane=controller-manager --tail=100 | \
  grep -i "azure\|error"
  2. Verify the SearchHeadCluster appSources configuration.
    CODE 
    kubectl -n $NAMESPACE get shc local-shc -o jsonpath='{.spec.appRepo.appSources[0].location}'
  3. Verify that the Azure blob exists.
    CODE 
    az storage blob list \
  --account-name splunkapps95484 \
  --container-name splunk-apps \
  --prefix localApps/