Upgrade the Splunk Operator using manifest files

Upgrade the Splunk Operator to the latest version using a manifest file.

Upgrading the Splunk Operator replaces the currently installed release with the latest version by applying an updated manifest file. Because the Splunk Operator does not support downgrading to a previous release, review your target version carefully before you upgrade.

Applying the latest splunk-operator-cluster.yaml or splunk-operator-namespace.yaml manifest updates the CRDs and Operator deployment, and applies the new spec to the pods the Operator manages. If the upgrade includes a newer Splunk Enterprise Docker image, the Operator also restarts those pods. For the restart order and behavior, see Splunk Operator upgrade behavior and order.

To upgrade using Helm instead, see Upgrade the Splunk Operator using Helm.

Prerequisites

  • Before you upgrade, review the Splunk Operator change log, which notes Splunk Enterprise Docker image compatibility for each release.
  • If the Docker image changes, review the Upgrade Readme and How to upgrade Splunk Enterprise for compatibility and upgrade guidance.
  • If you use forwarders, verify version compatibility in Compatibility between forwarders and Splunk Enterprise indexers.
  • Verify whether your manifest is cluster-wide or namespace-scoped by checking the WATCH_NAMESPACE environment variable in the running splunk-operator-controller-manager deployment. An empty value means splunk-operator-cluster.yaml is in use; a value naming one or more namespaces means splunk-operator-namespace.yaml is in use.

Apply the updated Splunk Operator manifest file

Upgrade the Splunk Operator by applying an updated manifest file.

Download the latest Splunk Operator installation YAML and apply it to upgrade your deployment.

  1. Download the latest Splunk Operator installation YAML file for your deployment type.

    Cluster-wide:

    CODE 
    wget -O splunk-operator-cluster.yaml https://github.com/splunk/splunk-operator/releases/download/<version>/splunk-operator-cluster.yaml

    Namespace scoped:

    CODE 
    wget -O splunk-operator-namespace.yaml https://github.com/splunk/splunk-operator/releases/download/<version>/splunk-operator-namespace.yaml
  2. Review the file for required updates, including accepting the Splunk General Terms, and add any customizations used during your original install.
    Note: The SPLUNK_GENERAL_TERMS environment variable is set to an empty string by default. You must manually update this to the required value. If you do not update this in the YAML file, you can update the splunk-operator-controller-manager deployment directly. See Splunk Operator upgrade configuration for details.
  3. Upgrade the Splunk Operator.

    Cluster-wide:

    CODE 
    kubectl apply -f splunk-operator-cluster.yaml --server-side

    Namespace scoped:

    CODE 
    kubectl apply -f splunk-operator-namespace.yaml --server-side

    After applying the YAML, Kubernetes creates a new operator pod and terminates the existing operator pod. Example:

    CODE 
    kubectl get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
splunk-operator-controller-manager-75f5d4d85b-8pshn   1/1     Running   0          5s

Verify the upgrade by checking the operator and Splunk Enterprise image versions.

To verify that the Splunk Operator upgrade applied the release image in splunk-operator-install.yaml, check the version of the operator image in the deployment spec and the image in the pod spec of the newly deployed operator pod:

CODE 
kubectl get deployment splunk-operator -o yaml | grep -i image
image: docker.io/splunk/splunk-operator:<desired_operator_version>
imagePullPolicy: IfNotPresent
CODE 
kubectl get pod <splunk_operator_pod> -o yaml | grep -i image
image: docker.io/splunk/splunk-operator:<desired_operator_version>
imagePullPolicy: IfNotPresent

To verify that a new Splunk Enterprise Docker image was applied to a pod, check the version of the image:

CODE 
kubectl get pods splunk-<crname>-monitoring-console-0 -o yaml | grep -i image
image: splunk/splunk:9.4.0
imagePullPolicy: IfNotPresent
Note: If a Splunk Operator release changes the custom resource (CRD) API version, you must update the Custom Resource specification to reference the latest version. For more information, see Splunk Operator upgrade behavior.

Splunk Operator upgrade configuration

Configuration requirements for Splunk Operator upgrades.

When you upgrade the Splunk Operator, review and update configuration for the Splunk General Terms acceptance and the namespaces the Operator watches.

Configure the operator to accept the Splunk General Terms

Starting with Operator version 3.0.0, which includes support for Splunk Enterprise version 10.x, an additional Docker-Splunk specific parameter is required to start containers.

Starting in 10.x image versions of Splunk Enterprise, license acceptance requires an additional SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com argument. This indicates that users have read and accepted the current version of the Splunk General Terms, available at Splunk General Terms.

Read and make sure you agree to the Splunk General Terms before you access or use this software. Only after doing so should you include the --accept-sgt-current-at-splunk-com flag to indicate your acceptance.

By default, the SPLUNK_GENERAL_TERMS environment variable is set to an empty string. You must either:

  • Manually update it to have the required value --accept-sgt-current-at-splunk-com in the splunk-operator-controller-manager deployment.
  • Pass the SPLUNK_GENERAL_TERMS parameter with the required value to the make deploy command.
  • Update the value in the Splunk Operator installation file from the release on GitHub.
CODE 
...
        env:
        - name: WATCH_NAMESPACE
          value: "splunk-operator"
        - name: RELATED_IMAGE_SPLUNK_ENTERPRISE
          value: splunk/splunk:9.4.0
        - name: OPERATOR_NAME
          value: splunk-operator
        - name: SPLUNK_GENERAL_TERMS
          value: "--accept-sgt-current-at-splunk-com"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
...

Configure the operator to watch a specific namespace

If the Splunk Operator is installed cluster-wide, edit the splunk-operator-controller-manager deployment in the splunk-operator namespace and set the WATCH_NAMESPACE field to the namespace that the Splunk Operator needs to monitor:

CODE 
...
        env:
        - name: WATCH_NAMESPACE
          value: "splunk-operator"
        - name: RELATED_IMAGE_SPLUNK_ENTERPRISE
          value: splunk/splunk:9.4.0
        - name: OPERATOR_NAME
          value: splunk-operator
        - name: SPLUNK_GENERAL_TERMS
          value: ""
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
...

If a Splunk Operator release includes an updated Splunk Enterprise Docker image, the operator upgrade also initiates pod restart using the latest Splunk Enterprise Docker image.

Splunk Operator upgrade behavior

Expected behavior and processing order when the Splunk Operator upgrades Splunk Enterprise deployments.

The Splunk Operator follows a defined behavior and processing order when it upgrades Splunk Enterprise deployments.

Upgrading the Splunk Enterprise Docker image with the Operator upgrade

The Splunk Operator follows the upgrade path described in the Splunk documentation, restarting pods with the latest Splunk Enterprise Docker image when a release includes one. For each existing CR instance, the Operator applies a specific flow to upgrade instances in order, verifies that each upgrade succeeded, and terminates the process if it detects unexpected behavior.

If a Splunk Operator release changes the custom resource (CRD) API version, the administrator is responsible for updating the Custom Resource specification to reference the latest CRD API version.

Splunk Enterprise cluster upgrade order

The Splunk Operator deletes and redeploys custom resource pods sequentially, redeploying multi-zone Indexer clusters zone by zone. For a Custom Resource with multiple pods, the Operator terminates and redeploys them in descending order, highest-numbered pod first.

When both the Operator version and the Splunk Enterprise Docker image are upgraded, pod termination and redeployment follow this order:

  1. Splunk Operator deployment pod
  2. Standalone
  3. License manager
  4. ClusterManager
  5. Search Head cluster
  6. Indexer Cluster
  7. Monitoring Console
Note: The order above assumes that the custom resources are linked through references. If there are Custom Resources without references, they are deleted and redeployed independently of the order.