Validation webhook

Intercept and validate custom resource specifications before they persist in the cluster.

The Splunk Operator includes an optional validation webhook that validates Splunk Enterprise custom resource specifications before the Kubernetes API server persists them.

The validation webhook intercepts CREATE and UPDATE operations on Splunk Enterprise CRDs and validates the spec fields according to predefined rules. If validation fails, the request is rejected with a descriptive error message.

Supported CRDs

The webhook validates the following Custom Resource Definitions:

  • Standalone
  • IndexerCluster
  • SearchHeadCluster
  • ClusterManager
  • LicenseManager
  • MonitoringConsole

Architecture

The validation webhook consists of the following components:

Component Description
Webhook server HTTP server listening on port 9443 with TLS
Validator registry Maps CRD types to their validation functions
ValidatingWebhookConfiguration Kubernetes resource that registers the webhook
Certificate TLS certificate managed by cert-manager
Service Kubernetes service exposing the webhook endpoint

Request flow

  1. A user submits a CREATE or UPDATE request for a Splunk CRD.
  2. The Kubernetes API server intercepts the request.
  3. The API server sends an AdmissionReview to the webhook service.
  4. The webhook server validates the spec fields.
  5. The webhook returns an Allowed or Denied response.
  6. If the webhook allows the request, the resource persists. If the webhook denies the request, the user receives an error.

Turn on the validation webhook

Turn om the validation webhook to validate Splunk Enterprise custom resource specifications before the Kubernetes API server persists them. The webhook is turned off by default and requires explicit opt-in.

Prerequisite

Install cert-manager in your cluster for TLS certificate management:

CODE 
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml
kubectl wait --for=condition=Available --timeout=300s deployment/cert-manager -n cert-manager
kubectl wait --for=condition=Available --timeout=300s deployment/cert-manager-webhook -n cert-manager

You can enable the validation webhook by using one of the following deployment options.

Option 1: Use the webhook-enabled Kustomize overlay

Deploy using the config/default-with-webhook overlay, which includes all webhook components:

CODE 
kustomize build config/default-with-webhook | kubectl apply -f -

Option 2: Enable the webhook on an existing deployment

If you already have the operator deployed, enable the webhook by setting the ENABLE_VALIDATION_WEBHOOK environment variable:

CODE 
kubectl set env deployment/splunk-operator-controller-manager \
  ENABLE_VALIDATION_WEBHOOK=true -n splunk-operator
Note: This option also requires the webhook service, ValidatingWebhookConfiguration, and TLS certificates. Use Option 1 for a complete deployment.

Option 3: Modify the default Kustomization

Edit config/default/kustomization.yaml to uncomment the webhook-related sections:

  1. Uncomment - ../webhook in the bases section.
  2. Uncomment - ../certmanager in the bases section.
  3. Uncomment - manager_webhook_patch.yaml in patchesStrategicMerge.
  4. Uncomment - webhookcainjection_patch.yaml in patchesStrategicMerge.
  5. Uncomment the vars section for certificate injection.

Then deploy:

CODE 
make deploy IMG=your-image SPLUNK_GENERAL_TERMS="--accept-sgt-current-at-splunk-com"

Validated fields: common fields for all CRDs

The validation webhook checks common and CRD-specific spec fields, including SmartStore and AppFramework configurations, when you submit custom resources.

Field Validation rule Error message
spec.etcVolumeStorageConfig.storageCapacity Must match format ^[0-9]+Gi$ (for example, "10Gi", "100Gi") must be in Gi format (e.g., '10Gi', '100Gi')
spec.varVolumeStorageConfig.storageCapacity Must match format ^[0-9]+Gi$ must be in Gi format (e.g., '10Gi', '100Gi')
spec.etcVolumeStorageConfig.storageClassName Required when ephemeralStorage=false and storageCapacity has a value storageClassName is required when using persistent storage
spec.varVolumeStorageConfig.storageClassName Required when ephemeralStorage=false and storageCapacity has a value storageClassName is required when using persistent storage

Validated fields: CRD-specific fields

CRD Field Validation rule
Standalone spec.replicas Must be greater than or equal to 0
IndexerCluster spec.replicas Must be greater than or equal to 3
SearchHeadCluster spec.replicas Must be greater than or equal to 3

Validated fields: SmartStore (Standalone and ClusterManager)

The webhook validates SmartStore configuration only when you provide it:

Field Validation rule
spec.smartstore.volumes[*].name Required (non-empty)
spec.smartstore.volumes[*] You must specify either endpoint or path
spec.smartstore.indexes[*].name Required (non-empty)
spec.smartstore.indexes[*].volumeName Required (non-empty)

Validated fields: AppFramework (Standalone, ClusterManager, and SearchHeadCluster)

The webhook validates AppFramework configuration only when you provide it:

Field Validation rule
spec.appRepo.appSources[*].name Required (non-empty)
spec.appRepo.appSources[*].location Required (non-empty)
spec.appRepo.volumes[*].name Required (non-empty)

Validation error examples

The following examples show common validation errors that the webhook returns when you submit invalid custom resource specifications.

Invalid replicas

A negative value for spec.replicas causes a validation error:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: example
spec:
  replicas: -1

Error:

CODE 
The Standalone "example" is invalid: .spec.replicas: Invalid value: -1: should be a non-negative integer

Invalid storage configuration

Using an incorrect storage capacity suffix causes a validation error:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: example
spec:
  etcVolumeStorageConfig:
    storageCapacity: "10GB"

Error:

CODE 
The Standalone "example" is invalid: spec.etcVolumeStorageConfig.storageCapacity: Invalid value: "10GB": must be in Gi format (e.g., '10Gi', '100Gi')

Missing SmartStore volume name

An empty name field for a SmartStore volume causes a validation error:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: example
spec:
  smartstore:
    volumes:
      - name: ""
        endpoint: "s3://bucket"

Error:

CODE 
The Standalone "example" is invalid: spec.smartstore.volumes[0].name: Required value: volume name is required

Verify webhook deployment

After you enable the validation webhook, verify that the webhook pod, TLS certificate, webhook registration, and operator logs indicate a successful deployment. Run the following commands to confirm that all webhook components are operational.

  1. Check that the webhook pod is running:
    CODE 
    kubectl get pods -n splunk-operator

    Expected output:

    CODE 
    splunk-operator-controller-manager-xxx   1/1   Running
  2. Check that the certificate is ready:
    CODE 
    kubectl get certificate -n splunk-operator

    Expected output:

    CODE 
    splunk-operator-serving-cert   True   webhook-server-cert
  3. Check that the webhook is registered:
    CODE 
    kubectl get validatingwebhookconfiguration splunk-operator-validating-webhook-configuration
  4. Check the operator logs for webhook startup messages:
    CODE 
    kubectl logs -n splunk-operator deployment/splunk-operator-controller-manager | grep -i webhook

    Look for the following messages:

    • Validation webhook enabled via ENABLE_VALIDATION_WEBHOOK=true
    • Starting webhook server {"port": 9443}

Troubleshoot the validation webhook

Diagnose and resolve common issues with the validation webhook, including registration failures, certificate problems, and configuration errors. The following issues can prevent the webhook from functioning correctly.

Issue: Webhook does not intercept requests

The webhook is not being called because the ValidatingWebhookConfiguration does not exist, the CA bundle is not injected, or the webhook service has no endpoints.

  1. Verify that the ValidatingWebhookConfiguration exists:
    CODE 
    kubectl get validatingwebhookconfiguration splunk-operator-validating-webhook-configuration -o yaml
  2. Check that the CA bundle is injected:
    CODE 
    kubectl get validatingwebhookconfiguration splunk-operator-validating-webhook-configuration \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | head -1

    The output should display -----BEGIN CERTIFICATE-----.

  3. Verify that the webhook service has endpoints:
    CODE 
    kubectl get endpoints -n splunk-operator splunk-operator-webhook-service

    The output should display an IP address.

Issue: Certificate problems

Certificate issues prevent the webhook from establishing TLS connections. The cert-manager might not have issued the certificate or the issuer might not exist.

  1. Check the cert-manager logs:
    CODE 
    kubectl logs -n cert-manager deployment/cert-manager
  2. Check the certificate status:
    CODE 
    kubectl describe certificate -n splunk-operator splunk-operator-serving-cert
  3. Check the issuer:
    CODE 
    kubectl get issuer -n splunk-operator

Issue: Webhook is not enabled

The webhook is turned off because the ENABLE_VALIDATION_WEBHOOK environment variable is not set to true, or the deployment does not use the webhook-enabled Kustomize overlay.

  1. Verify that the ENABLE_VALIDATION_WEBHOOK environment variable is set to true.
  2. Verify that you deployed with the correct Kustomize overlay (config/default-with-webhook).

Turn off the validation webhook

Turn off the validation webhook by updating the environment variable or by redeploying without the webhook overlay. Use one of the following methods.

Set the environment variable to false

CODE 
kubectl set env deployment/splunk-operator-controller-manager \
  ENABLE_VALIDATION_WEBHOOK=false -n splunk-operator

Redeploy without the webhook overlay

Redeploy using the default Kustomization, which does not include the webhook:

CODE 
make deploy IMG=your-image SPLUNK_GENERAL_TERMS="--accept-sgt-current-at-splunk-com"