Set up the LOCAL cluster for federated search

Configure the LOCAL search head cluster to search REMOTE indexes.

Complete the steps in Set up the REMOTE cluster for federated search before you continue.

The LOCAL cluster is where searches run. Create the federated provider and index configuration files, then package and deploy the configuration in an app to the cluster.

  1. Create the federated search app structure.
    CODE 
    mkdir -p federated-search-app/default
mkdir -p federated-search-app/metadata
  2. Create app.conf.
    CODE 
    cat > federated-search-app/default/app.conf << 'EOF'
[install]
is_configured = 1
state = enabled

[ui]
is_visible = 0
label = Federated Search Configuration

[launcher]
author = Splunk Admin
description = Federated search provider and index configurations
version = 1.0.0
EOF
  3. Create federated.conf to define the connection to the REMOTE cluster.
    CODE 
    cat > federated-search-app/default/federated.conf << 'EOF'
# Federated Provider Configuration
[provider://remote_thru_nginx]
appContext = search
hostPort = remote-mgmt.stos-auto.svc.cluster.local:443
mode = standard
password = SvcP@ssw0rd
serviceAccount = fsh_svc
type = splunk
useFSHKnowledgeObjects = 0
EOF

    Configuration parameters:

    appContext
    The app context on the remote cluster. Usually search.
    hostPort
    The remote cluster management endpoint, as FQDN:port.
    mode
    standard for search-to-search, or transparent for forwarding.
    password
    The service account password.
    serviceAccount
    The username on the remote cluster.
    type
    Always splunk for Splunk-to-Splunk federation.
    useFSHKnowledgeObjects
    Whether to use remote knowledge objects. 0 is false, 1 is true.
  4. Create indexes.conf to define the virtual federated indexes.
    CODE 
    cat > federated-search-app/default/indexes.conf << 'EOF'
# Federated Index Configuration - Maps to REMOTE _audit index
[federated:r_audit]
federated.provider = remote_thru_nginx
federated.dataset = index:_audit

# Federated Index Configuration - Maps to REMOTE demo index
[federated:r_demo]
federated.provider = remote_thru_nginx
federated.dataset = index:demo
EOF

    Use the naming convention federated:r_indexname for federated indexes. The r_ prefix identifies the index as federated ("remote") so it's clear which indexes are federated and which are local. federated.provider references the provider name from federated.conf, and federated.dataset specifies the remote index as index:remote_index_name.

  5. Create default.meta.
    CODE 
    cat > federated-search-app/metadata/default.meta << 'EOF'
[]
access = read : [ * ], write : [ admin ]
export = system
EOF
  6. Set the LOCAL pod and credentials.
    CODE 
    LOCAL_POD=$(kubectl -n $NAMESPACE get pods \
  -l app.kubernetes.io/instance=splunk-local-shc-search-head \
  -o jsonpath='{.items[0].metadata.name}')
LOCAL_ADMIN=$(kubectl -n $NAMESPACE get secret splunk-local-shc-search-head-secret-v1 \
  -o jsonpath='{.data.password}' | base64 -d)

    The built-in admin role already includes the search capability. If you validate with a custom local role, verify that it has search plus access to the federated indexes defined in this app.

  7. Package and deploy the app using one of the following options.
    Option A: App Framework on Azure (validated path)

    This is the Azure-specific path used for the SVA C3 deployment. If you use S3 or Google Cloud Storage instead, adapt the appRepo storage settings as described in App Framework. If you don't want to use a remote app repository, use option B instead.

    Package the app and upload it to Azure Storage:

    CODE 
    # Package the app
cd federated-search-app
tar czf ../federated-search-app_1.0.0.tgz .
cd ..

# Upload to Azure Storage (localApps path for LOCAL SHC)
az storage blob upload \
  --account-name splunkapps95484 \
  --container-name splunk-apps \
  --name "localApps/federated-search-app_1.0.0.tgz" \
  --file federated-search-app_1.0.0.tgz \
  --overwrite

    Create or update the LOCAL search head cluster with the App Framework configuration:

    CODE 
    # local-shc-appframework.yaml
apiVersion: enterprise.splunk.com/v4
kind: SearchHeadCluster
metadata:
  name: local-shc
  namespace: stos-auto
spec:
  replicas: 3
  clusterManagerRef:
    name: local-cm
  serviceAccount: splunk-operator-sa
  appRepo:
    appsRepoPollInterval: 300
    defaults:
      volumeName: volume_app_repo
      scope: cluster
    appSources:
      - name: localApps
        location: localApps/
    volumes:
      - name: volume_app_repo
        storageType: azure
        provider: azure
        azureSecretRef: azure-blob-secret
        path: splunkapps95484/splunk-apps/

    Apply the configuration:

    CODE 
    kubectl apply -f local-shc-appframework.yaml
    Option B: Manual deployment

    Copy the app directly to all LOCAL search head cluster pods:

    JSON 
    for pod in $(kubectl -n $NAMESPACE get pods \
  -l app.kubernetes.io/instance=splunk-local-shc-search-head \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n $NAMESPACE cp -c splunk federated-search-app \
    "$pod:/opt/splunk/etc/apps/"
done

# Restart Splunk on all pods
kubectl -n $NAMESPACE delete pod -l app.kubernetes.io/instance=splunk-local-shc-search-head
  8. Wait for deployment to complete.

    If you used App Framework, App Framework polls every 5 minutes by default. Monitor the operator logs and wait for the search head cluster to stabilize:

    CODE 
    # Monitor the operator logs
kubectl -n splunk-operator logs -l control-plane=controller-manager -f | \
  grep -i "appframework\|download"

# Wait for SHC to stabilize
kubectl -n $NAMESPACE wait --for=condition=Ready --timeout=600s \
  pod -l app.kubernetes.io/instance=splunk-local-shc-search-head
  9. Verify the LOCAL configuration.
    CODE 
    # Check federated provider configuration
kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- \
  /opt/splunk/bin/splunk btool federated list provider://remote_thru_nginx

# Check federated index configuration
kubectl -n $NAMESPACE exec $LOCAL_POD -c splunk -- \
  /opt/splunk/bin/splunk btool indexes list federated:r_audit
    The provider command shows all provider settings. The index command returns:
    CODE 
    [federated:r_audit]
federated.dataset = index:_audit
federated.provider = remote_thru_nginx