Replace a Node in the Control Plane

Replace a failed Virtual Appliance cluster node in the control plane and restore high availability.

If a control plane node fails and cannot be recovered, Virtual Appliance performance may be impacted. Use the following steps to replace the node and restore system performance:

Important Considerations

Persistent data stored on the failed node's local disk is not automatically migrated to the replacement node. Kubernetes reschedules pods to healthy nodes, but stateful workloads (databases, Elasticsearch) rely on their own replication mechanisms for data availability.

  1. Identify the failed control plane node.
    1. Run the following command to view the cluster nodes.
      CODE 
      appdctl show cluster

      Sample Output:

      CODE 
      NODE           | ROLE  | RUNNING 
----------------+-------+---------
 10.0.0.1:19001 | voter | true    
 10.0.0.2:19001 | voter | true    
 10.0.0.3:19001 | voter | true
      Note: The output may continue to show RUNNING: true even if the control plane node is failed. Therefore, you must confirm the node status using microk8s kubectl get nodes.
    2. Verify the Kubernetes node status.
      CODE 
      microk8s kubectl get nodes

      Sample Output:

      CODE 
      NAME       STATUS     ROLES    AGE   VERSION
node-1     Ready      master   30d   v1.34.8
node-2     Ready      master   30d   v1.34.8
node-3     NotReady   master   30d   v1.34.8

      In this example, node-3 has failed and requires replacement.

  2. Remove the failed node from the cluster:
    CODE 
    microk8s remove-node <node-ip-address> --force

    For example:

    CODE 
    microk8s remove-node 10.0.0.3 --force
    Note: After removing a node, the cluster temporarily loses high availability. The microk8s status output shows high-availability: no until a replacement node is added.
    1. Verify whether the node is removed.
      CODE 
      appdctl show cluster

      Sample Output:

      CODE 
      NODE               | ROLE  | RUNNING
--------------------+-------+---------
 10.0.0.1:19001     | voter | true
 10.0.0.2:19001     | spare | true
    2. Verify the Kubernetes node status.
      CODE 
      microk8s kubectl get nodes

      Sample Output:

      CODE 
      NAME       STATUS   ROLES    AGE   VERSION
node-1     Ready    master   30d   v1.34.8
node-2     Ready    master   30d   v1.34.8
  3. To initialize a replacement node, deploy the Virtual Appliance image version as the existing cluster:
    1. Deploy a node using the same Virtual Appliance version as the existing version. See Deploy Splunk AppDynamics On-Premises Virtual Appliance.
    2. Verify the boot status of the replacement node:
      CODE 
      appdctl show boot

      Sample Output:

      CODE 
      NAME              | STATUS    | ERROR
-------------------+-----------+-------
 cloud-init-config | Succeeded | --
 enable-time-sync  | Succeeded | --
 firewall-setup    | Succeeded | --
 storage-setup     | Succeeded | --
 hostname          | Succeeded | --
 microk8s-setup    | Succeeded | --
 netplan           | Succeeded | --
 ssh-setup         | Succeeded | --
 cert-setup        | Succeeded | --
      Note:

      Ensure that all services appear as Succeeded. If any service appears as Failed, restart the virtual machine. If it still fails, redeploy the virtual machine.

  4. Add the replacement node to the cluster.
    1. Run the following command on a healthy cluster node to generate the join token:
      CODE 
      appdctl cluster add-node

      Sample Output:

      CODE 
      Enter the following command on the node as master to join the cluster:
appdctl cluster join 10.0.0.1:25000/0b0e24950c829e2df614a1d8866d703f/0950dca162dd

Enter one of the following command to join the node as a worker:
appdctl cluster join 10.0.0.1:25000/0b0e24950c829e2df614a1d8866d703f/0950dca162dd --worker
microk8s.join 10.0.0.1:25000/0b0e24950c829e2df614a1d8866d703f/0950dca162dd --worker
    2. Run the following command to join the replacement node to the cluster's control plane.
      CODE 
      appdctl cluster join <copy-the-generated-join-token>

      For example:

      CODE 
      appdctl cluster join 10.0.0.1:25000/0b0e24950c829e2df614a1d8866d703f/0950dca162dd

      Sample Output:

      JSON 
      {
    "status": "WARNING: Hostpath storage is enabled and is not suitable for multi node clusters.\n\nContacting cluster at 10.0.0.1\nWaiting for this node to finish joining the cluster. .. .. .. .. ..  \nSuccessfully joined the cluster.\n"
}
      CAUTION: Do not use the --worker flag when replacing a failed node. The replacement node must join the control plane to restore high availability and dqlite quorum.
    3. Verify the cluster includes the replacement node:
      CODE 
      appdctl show cluster

      Sample Output:

      CODE 
      NODE               | ROLE  | RUNNING
--------------------+-------+---------
 10.0.0.1:19001     | voter | true
 10.0.0.2:19001     | voter | true
 10.0.0.4:19001     | voter | true
    4. Verify the Kubernetes node status.
      CODE 
      microk8s kubectl get nodes

      Sample Output:

      CODE 
      NAME       STATUS   ROLES    AGE     VERSION
node-1     Ready    master   30d     v1.34.8
node-2     Ready    master   30d     v1.34.8
node-4     Ready    master   2m      v1.34.8
    Once the replacement node successfully joins the cluster, it initiates data replication from existing nodes.
  5. Verify the data replication status using the following command:
    CODE 
    appdcli run infra_inspect

    Sample Output:

    CODE 
    NAME                                READY   STATUS      RESTARTS   AGE
appd-mysqlsh-0                      1/1     Running     0          4m33s
appd-mysql-0                        2/2     Running     0          4m33s
appd-mysql-1                        2/2     Running     0          4m33s
appd-mysql-2                        2/2     Running     0          4m33s
appd-mysql-router-9f8bc6784-g7zx7   1/1     Running     0          5s
appd-mysql-router-9f8bc6784-fhjnp   1/1     Running     0          5s
appd-mysql-router-9f8bc6784-wrcwk   1/1     Running     0          5s
  6. Verify whether the high-availability status is restored.
    CODE 
    microk8s status

    Sample Output:

    CODE 
    microk8s is running
high-availability: yes
  datastore master nodes: 10.0.0.1:19001 10.0.0.2:19001 10.0.0.4:19001
  datastore standby nodes: none