Add or Remove Worker Nodes of the Cluster

The Virtual Appliance infrastructure services might run out of resources when handling large workloads. In these cases, adding an extra node to the cluster can help the infrastructure services to manage the increased workloads.
Note: These procedures apply to Virtual Appliance deployments. They do not apply to the deployments using custom Kubernetes.

The appdctl cluster add-workers and appdctl cluster remove-workers commands change cluster infrastructure and Virtual Appliance metadata. They do not add or remove AppDynamics application-service capacity. Scale application services separately when applicable.

If a worker node in the cluster fails, replace the failed node as follows:
  1. Deploy and bootstrap a replacement worker with the same Virtual Appliance version and build as the control-plane nodes.
  2. Remove the failed worker node. See Remove Worker Nodes.
  3. Add and join a new worker node to the cluster. See Add a Worker Node.
  4. Verify the cluster, then scale out the required application service to the replacement worker.

Add a Worker Node

Complete the following steps on an existing cluster-plane node:
  • Log in as appduser on an existing, healthy control-plane node.
  • Wait for any cluster upgrade to complete successfully.
  • Deploy and bootstrap each worker with the same Virtual Appliance version and build as the control-plane nodes.
  • Size each worker for the installed standard or hybrid profile.
  • Verify that Kubernetes does not list any worker as a node and that each worker has a unique InternalIP.
  • Verify SSH and SCP connectivity from the control-plane node to worker node.
  • Verify that /home/appduser/.ssh/id_rsa.pub exists on the control-plane node.
  1. Check the cluster health.
    CODE 
    appdctl show cluster
microk8s status --wait-ready
microk8s kubectl get nodes -o wide
microk8s kubectl get pods -A -o wide
    Resolve unhealthy nodes or pods before you continue.
  2. Add one or more workers.

    To add one worker, run:

    CODE 
    appdctl cluster add-workers --peers <worker-internal-IP>

    To add multiple workers, run:

    CODE 
    appdctl cluster add-workers --peers <worker-1-internal-IP>,<worker-2-internal-IP>
  3. Verify that every added worker reports Ready and that the cluster and pods report healthy status.
    CODE 
    microk8s status --wait-ready
microk8s kubectl get nodes -o wide
microk8s kubectl get nodes --show-labels
appdctl show cluster
microk8s kubectl get pods -A -o wide
  4. Optional: If you added the worker to increase Elasticsearch service capacity, scale out Elasticsearch.
    CODE 
    appdcli platform scale out --node <worker-node-name-or-internal-IP> --service elasticsearch
    Repeat --node or use a comma-separated value for multiple ready, schedulable workers. Skip this step if you do not want to assign supported application-service capacity to the new worker. For more information, see Scaling Elasticsearch Capacity on Virtual Appliance.

Remove Worker Nodes

You must remove the worker node from an existing control plane node. Perform the following steps to remove one or more worker nodes:

  1. Remove the worker nodes.
    Single node
    CODE 
    appdctl cluster remove-workers --peers <internal-IP-address>
    Multiple nodes
    CODE 
    appdctl cluster remove-workers --peers <internal-IP-address>, <interal-IP-address>

    This command also cleans the files related to the removed worker nodes.

    Note: Use manual cleanup only when the command reports that automatic cleanup was skipped or failed.
    CODE 
    sudo microk8s leave || true
sudo rm -f /var/appd-os/cluster-info.yaml
if [ -f /etc/hosts.orig ]; then sudo cp /etc/hosts.orig /etc/hosts; fi
rm -f /home/appduser/.kube/config /home/appduser/cluster.yaml
  2. Verify the removed worker nodes.
    CODE 
    microk8s status --wait-ready
microk8s kubectl get nodes -o wide
appdctl show cluster
microk8s kubectl get pods -A -o wide