Scaling Elasticsearch Capacity on Virtual Appliance

Change the Elasticsearch replica count in a standard Virtual Appliance deployment.

The Virtual Appliance supports Elasticsearch capacity scale up or scale down to optimize system performance. Capacity scale up increases the replica count in your Virtual Appliance. During scaling, Elastic Cloud on Kubernetes (ECK) can redistribute existing shards across the resulting replicas.

Scaling Virtual Appliance can handle higher workloads and reduces performance bottlenecks. Scale the Elasticsearch capacity only after adding worker node to the cluster.

Prerequisites

Complete the following checks before scaling the Elasticsearch capacity:

  • Run the commands as the appduser user from a Virtual Appliance node that has access to appdcli, Kubernetes, and the deployment configuration.

  • Verify whether the target node has joined the cluster, reports Ready, and remains schedulable.

  • Complete any cluster changes or upgrade.

  • Ensure there are no workloads running on the target node.

  • Confirm green ECK health and confirm that no earlier scale operation continues shard relocation or recovery.
  • Confirm that the storage class can provision an additional persistent volume for each replica.
  • Verify that you can decrypt the Helm secrets.
    CODE 
    helm secrets decrypt /var/appd/config/secrets.yaml.encrypted >/dev/null
CAUTION: Do not run this command during a Virtual Appliance cluster upgrade or concurrently with another Helm operation.

Scale Up the Elasticsearch Capacity

Complete the following steps to scale the Elasticsearch capacity on a worker node:
Confirm the following requirements:
  • Elasticsearch cluster is healthy.
  • No shard relocation or recovery are in progress.

  • There is enough space and CPU available on the target node.

  • The elastic operator and Elasticsearch cluster are healthy

  1. Log in to the cluster node that has access to appdcli and Helm.
  2. Check the cluster before scaling:
    CODE 
    kubectl get elasticsearch appd -n es
kubectl get nodes -o wide
kubectl get pods -n es -o wide
kubectl get pvc -n es
  3. Scale out the Elasticsearch capacity.
    Single Node
    CODE 
    appdcli platform scale out --service es --node <node-name-or-internal-ip>
    Example:
    CODE 
    appdcli platform scale out --service es --node 10.0.0.1
    Multiple Nodes
    CODE 
    appdcli platform scale out --service es --node <node-name-or-internal-ip> --node <node-name-or-internal-ip>
    Example:
    CODE 
    appdcli platform scale out --service es --node 10.0.0.1 --node 10.0.0.2
    Sample Output:
    JAVASCRIPT 
    Elasticsearch scale-out capacity preflight passed for ip-10-115-86-115: requires 100m CPU and 1Gi memory per pod
Updated service scaling state in /var/appd/config/scaling-overlay.yaml.gotmpl
Action: scale-out
Services: elasticsearch
Elasticsearch replicas: 3 -> 4
Elasticsearch scale-out plan:
  Target replicas: 3 -> 4
  Pod(s) expected to be added: appd-es-node-3
  Data movement: Elasticsearch/ECK may rebalance shards after the new pod joins.
Using install profile: small
Helmfile progress
  Full logs: /tmp/helmfile-progress.NfmwXQ.log
  Helmfile: /home/appduser/appd-charts/helmfile.yaml.gotmpl

Overall: 0/50 complete | Success: 0 | Failed: 0 | Running: 0 | Pending: 50

[Setup]
  … Prepare: Initializing Helm prerequisites
  ✓ Prepare: Environment files and generated values are ready

[Preflight]
  … Disk health: Checking kubelet filesystem usage and disk pressure
  ✓ Disk health: Disk health status is HEALTHY
  … Terminating pods: Checking AppD namespaces for blocking terminating pods
  ✓ Terminating pods: No blocking terminating pods found

[Infra]
  … Elasticsearch
  ✓ Elasticsearch
Overall: 1/50 complete | Success: 1 | Failed: 0 | Running: 0 | Pending: 49
Phase: Infra 1/16 complete | Failed: 0

Helmfile summary
  Completed successfully
  Time taken: 14s
  Full logs: /tmp/helmfile-progress.NfmwXQ.log
Elasticsearch shard movement and allocation monitoring continues in background. Detailed log: /home/appduser/appdcli_elasticsearch_scale_output.log
To follow progress: tail -f /home/appduser/appdcli_elasticsearch_scale_output.log

    The command increases the replica count by the number of unique resolved nodes.

  4. Optional: Run this command to follow the scaling progress:
    CODE 
    tail -f /home/appduser/appdcli_elasticsearch_scale_output.log
  5. Verify replica count, pod health, placement, and storage.
    CODE 
    kubectl get elasticsearch appd -n es
kubectl get pods -n es -o wide
kubectl get pvc -n es

    Confirm that the pod count matches the target replica count.

Scale Down the Elasticsearch Capacity

If Virtual Appliance performance remains optimal and a resource decrease does not impact system stability, perform an Elasticsearch capacity scale down. This process removes Elasticsearch replicas before you remove workers from a standard Virtual Appliance cluster.

Confirm the following requirements:

  • Elasticsearch cluster is healthy.
  • No shard relocation or recovery are in progress.
  • Back up the or snapshot the node data.
  • Capacity reduction is safe for the existing data volume.
Note: Do not remove a worker until the scale-in monitor completes.
  1. Check the existing Elasticsearch pods and their storage.
    CODE 
    kubectl get pods -n es -o wide
kubectl get pvc -n es
kubectl get pv
  2. Run the following command to scale down the Elasticsearch capacity.
    CODE 
    appdcli platform scale in --service es --node <node-name-or-internal-ip>
    Sample Output:
    JAVASCRIPT 
    Updated service scaling state in /var/appd/config/scaling-overlay.yaml.gotmpl
Action: scale-in
Services: elasticsearch
Elasticsearch replicas: 4 -> 3
Elasticsearch scale-in plan:
  Target replicas: 4 -> 3
  Pod(s) expected to be removed: appd-es-node-3
  PVC(s) expected to become stale: elasticsearch-data-appd-es-node-3
  Data movement: Elasticsearch/ECK relocates shard data before pod removal; appdcli cleans stale storage afterward.
Using install profile: small
Helmfile progress
  Full logs: /tmp/helmfile-progress.q4yHBO.log
  Helmfile: /home/appduser/appd-charts/helmfile.yaml.gotmpl

Overall: 0/50 complete | Success: 0 | Failed: 0 | Running: 0 | Pending: 50

[Setup]
  … Prepare: Initializing Helm prerequisites
  ✓ Prepare: Environment files and generated values are ready

[Preflight]
  … Disk health: Checking kubelet filesystem usage and disk pressure
  ✓ Disk health: Disk health status is HEALTHY
  … Terminating pods: Checking AppD namespaces for blocking terminating pods
  ✓ Terminating pods: No blocking terminating pods found

[Infra]
  … Elasticsearch
  ✓ Elasticsearch
Overall: 1/50 complete | Success: 1 | Failed: 0 | Running: 0 | Pending: 49
Phase: Infra 1/16 complete | Failed: 0

Helmfile summary
  Completed successfully
  Time taken: 13s
  Full logs: /tmp/helmfile-progress.q4yHBO.log
Elasticsearch shard movement and allocation monitoring continues in background. Detailed log: /home/appduser/appdcli_elasticsearch_scale_output.log
To follow progress: tail -f /home/appduser/appdcli_elasticsearch_scale_output.log
Stale Elasticsearch PVC/PV cleanup will run from the background monitor after the removed pod is gone and Elasticsearch is settled.
  3. Optional: Run this command to follow the scaling progress:
    CODE 
    tail -f /home/appduser/appdcli_elasticsearch_scale_output.log

    Wait for elasticsearch scale-in monitor completed.

  4. Verify the resulting replicas and storage.
    CODE 
    kubectl get elasticsearch appd -n es
kubectl get pods -n es -o wide
kubectl get pvc -n es
kubectl get pv

    Confirm green health, the target replica count, absence of the removed highest-ordinal pods and stale PVCs, and absence of released PVs for the removed PVCs.

  5. Verify whether the stale storage is cleaned.
    CODE 
    kubectl get pvc -n es
kubectl get pv
After scale down of Elasticsearch capacity, verify these items:
  • Elasticsearch pod count matches the target replica count.
  • Removed pod no longer exists in cluster.
  • Stale PVCs with ordinals greater than or equal to target replica count undergo removal.
  • Released stale persistent volumes for the removed PVCs.