Back up and restore using NFS

Follow the steps to back up and restore using NFS.

Set Up the NFS Server

Install and configure an NFS server on AlmaLinux or Ubuntu for Virtual Appliance backups.

Follow these steps to install and configure the NFS server on AlmaLinux or Ubuntu:

  1. Run the following commands to install the NFS server.
    AlmaLinux
    CODE 
    sudo dnf install nfs-utils
sudo mkdir -p /mnt/nfs_share
sudo chown nobody:nobody /mnt/nfs_share
sudo chmod 777 -R /mnt/nfs_share/
    Ubuntu
    CODE 
    sudo apt install nfs-kernel-server
sudo mkdir -p /mnt/nfs_share
sudo chown nobody:nogroup /mnt/nfs_share
sudo chmod 777 -R /mnt/nfs_share/
  2. Configure the NFS exports.
    CODE 
    /mnt/nfs_share 10.0.203.0/24(rw,sync,no_subtree_check,no_root_squash)
/mnt/nfs_share 10.0.202.0/24(rw,sync,no_subtree_check,no_root_squash)
  3. Restart the NFS server:
    CODE 
    sudo systemctl restart nfs-server
  4. Verify the NFS server status:
    CODE 
    sudo systemctl status nfs-server

Back Up Elasticsearch to NFS Storage

Create and verify an Elasticsearch snapshot in an NFS-backed repository.

Configure the NFS backup backend and make the repo3 repository available.
  1. Create the NFS snapshot repository.
    CODE 
    appdcli backup elasticsearch create-fs-repo repo3
  2. Verify that repo3 is listed as an fs repository.
    CODE 
    appdcli backup elasticsearch list-repos
  3. Create a snapshot.
    CODE 
    appdcli backup elasticsearch create-snapshot repo3 <snapshot-name>
  4. Wait until the snapshot state is SUCCESS.
    CODE 
    appdcli backup elasticsearch list-snapshots repo3
The Elasticsearch snapshot is stored in the NFS-backed repository.

Restore Elasticsearch from NFS Storage

Rebuild Elasticsearch data PVCs and restore a snapshot from an NFS-backed repository.

Record the snapshot name and confirm that the NFS backup PVC is healthy.
Warning: This procedure stops the events service and Elasticsearch.
CAUTION: Delete only the Elasticsearch data PVCs. Do not delete the NFS backup PVC, such as es-backup-pvc.
  1. Scale the events service to zero replicas and wait for its pods to terminate.
    CODE 
    kubectl scale deploy events-ss -n cisco-events --replicas=0
  2. Remove the cluster UUID annotation and stop the Elasticsearch StatefulSets.
    CODE 
    kubectl annotate elasticsearch appd -n es "elasticsearch.k8s.elastic.co/cluster-uuid-"
kubectl scale statefulset appd-es-node -n es --replicas=0
kubectl scale statefulset es-shell -n es --replicas=0
  3. List and identify the Elasticsearch data PVCs.
    CODE 
    kubectl get pvc -n es
  4. Delete the Elasticsearch data PVCs, and then verify that only the backup PVC remains.
    CODE 
    kubectl delete pvc <elasticsearch-data-pvc-0> <elasticsearch-data-pvc-1> <elasticsearch-data-pvc-2> -n es
kubectl get pvc -n es
  5. Start the Elasticsearch StatefulSets and wait until the cluster is ready.
    CODE 
    kubectl scale statefulset appd-es-node -n es --replicas=3
kubectl scale statefulset es-shell -n es --replicas=3
kubectl get elasticsearch -n es

    Continue when the health is green and the phase is Ready.

  6. Restore the snapshot.
    CODE 
    appdcli backup elasticsearch restore-snapshot repo3 <snapshot-name>
  7. Start the events service.
    CODE 
    kubectl scale deploy events-ss -n cisco-events --replicas=3
Elasticsearch data is restored and the events service is running.

Back Up PostgreSQL to NFS Storage

Create and verify a PostgreSQL backup in the NFS-backed pgBackRest repository.

Configure repo3 as the NFS-backed pgBackRest repository.
  1. Create a backup.
    CODE 
    appdcli backup postgres create-snapshot repo3

    Record the generated perconapgbackup resource name.

  2. Wait until the backup status is Succeeded.
    CODE 
    kubectl get perconapgbackup <backup-resource-name> -n postgres
  3. List the backups recorded in the repository.
    CODE 
    appdcli backup postgres list-snapshots repo3

Restore PostgreSQL from NFS Storage

Restore the latest PostgreSQL backup in place from the NFS-backed pgBackRest repository.

The command restores one instance and automatically recycles replica pods and PVCs that have a PostgreSQL system-ID mismatch.
  1. List available backups and review the RESTORE column.
    CODE 
    appdcli backup postgres list-snapshots repo3

    If the repository lineage has changed, use --set=<label> with the restore command.

  2. Restore the backup and confirm the prompt.
    CODE 
    appdcli backup postgres restore-snapshot repo3
    Note: Use the --force option only when you intend to skip the confirmation prompt.
  3. Wait until the restore status is Succeeded.
    CODE 
    kubectl get perconapgrestore -n postgres
  4. Verify that all PostgreSQL instances are ready.
    CODE 
    kubectl get perconapgcluster -n postgres

Recover PostgreSQL from NFS Storage After PVC Loss

Rebuild PostgreSQL instance PVCs and restore a labeled backup from NFS storage.

Record the label of a backup taken before the PVC loss.
Warning: Pausing the cluster stops PostgreSQL.
CAUTION: Delete only PostgreSQL instance data PVCs. Do not delete a backup repository PVC, such as appd-postgres-repo1.
  1. Pause the cluster and wait until all instance pods terminate.
    JSON 
    kubectl patch perconapgcluster appd-postgres -n postgres --type=merge -p '{"spec":{"pause":true}}'
kubectl get pods -n postgres -l postgres-operator.crunchydata.com/instance-set=instance
  2. List and delete the lost or corrupted instance PVCs.
    CODE 
    kubectl get pvc -n postgres
kubectl delete pvc <postgres-instance-pvc-1> <postgres-instance-pvc-2> <postgres-instance-pvc-3> -n postgres
  3. Unpause the cluster and wait for new PVCs and pods.
    JSON 
    kubectl patch perconapgcluster appd-postgres -n postgres --type=merge -p '{"spec":{"pause":false}}'
kubectl get pods -n postgres -w
  4. List backups and identify the pre-loss label.
    CODE 
    appdcli backup postgres list-snapshots repo3
  5. Restore the labeled backup.
    CODE 
    appdcli backup postgres restore-snapshot repo3 --set=<backup-label>
  6. Verify the restore and replication health.
    CODE 
    kubectl get perconapgrestore -n postgres
kubectl exec -n postgres <primary-pod> -c database -- patronictl list

    Verify a running leader and streaming replicas.

Back Up MySQL to NFS Storage

Create a full MySQL XtraBackup chain in NFS storage.

Configure MySQL XtraBackup to use the NFS backend.
  1. Create a full backup to start a new chain.
    CODE 
    appdcli backup mysql full

    Use the same MySQL pod for a full backup and all incremental backups in its chain.

  2. Wait until the XtraBackup phase is Completed.
    CODE 
    appdcli backup mysql list
  3. List backup chains and review the generated listing job.
    CODE 
    appdcli backup mysql list-chains
kubectl logs -n mysql -l job-name=<xtrabackup-listing-job>

Restore MySQL from NFS Storage

Stop the MySQL cluster, restore an XtraBackup chain from NFS storage, and rebuild the InnoDB Cluster.

List backup chains and record the chain timestamp to restore.
Warning: This procedure causes a full MySQL outage. Run the commands in order and wait for each stated condition.
  1. Reduce the cluster to one instance, stop the operator, and stop the StatefulSet.
    JSON 
    kubectl patch innodbcluster appd-mysql -n mysql --type=merge -p '{"spec":{"instances":1}}'
kubectl scale deploy mysql-operator -n mysql-operator --replicas=0
kubectl scale sts appd-mysql -n mysql --replicas=0

    Wait until the MySQL pods show Completed.

  2. Restore the selected backup chain and confirm the prompt.
    CODE 
    appdcli backup mysql restore --from-chain <chain-timestamp>
  3. Wait for the XtraRestore phase to show Completed.
    CODE 
    kubectl get xtrarestore <xtrarestore-name> -n mysql
  4. Clear the first pod's finalizer and start one MySQL instance while the operator remains stopped.
    JSON 
    kubectl patch pod appd-mysql-0 -n mysql -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl scale sts appd-mysql -n mysql --replicas=1
  5. Reset the metadata schema and create a single-node InnoDB Cluster.
    CODE 
    kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh --sql -u root -p'<root-password>' -e "SET GLOBAL super_read_only=OFF; SET GLOBAL read_only=OFF; DROP DATABASE IF EXISTS mysql_innodb_cluster_metadata;"
kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh -u root -p'<root-password>' -- dba create-cluster appd-mysql --force
  6. Start the operator and return the cluster to three instances.
    JSON 
    kubectl scale deploy mysql-operator -n mysql-operator --replicas=1
kubectl patch innodbcluster appd-mysql -n mysql --type=merge -p '{"spec":{"instances":3}}'
kubectl get innodbcluster appd-mysql -n mysql

    Wait for the status to show ONLINE with three online instances.

  7. Recycle the MySQL Router pods.
    CODE 
    kubectl delete pod -n mysql -l component=mysqlrouter

Recover MySQL Replicas After an NFS Restore

Join MySQL replicas manually when PVC loss prevents automatic recovery after an NFS restore.

Use this procedure when the restored cluster does not reach ONLINE and an affected replica reports Shell Error (51113): Metadata Schema not found.
  1. Delete the affected replica pod and its data PVC.
    CAUTION: Deleting the PVC permanently removes that replica's data. Verify the health of the restored primary first.
    CODE 
    kubectl delete pod <mysql-replica-pod> -n mysql --force --grace-period=0
kubectl delete pvc <mysql-replica-pvc> -n mysql
  2. Wait for the StatefulSet to create a new pod and PVC.
    CODE 
    kubectl get pods -n mysql -l component=mysqld
  3. From appd-mysql-0, add the instance with clone-based recovery.
    JSON 
    kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh --py -u root -p'<root-password>' -e "
cluster = dba.get_cluster('appd-mysql')
cluster.add_instance({'host': '<replica-host>', 'port': 3306, 'user': 'root', 'password': '<root-password>'}, {'recoveryMethod': 'clone'})
"
  4. Repeat the recovery for each affected replica and verify an ONLINE state for every member.
    CODE 
    kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh --sql -u root -p'<root-password>' -e "SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;"
  5. Recycle the MySQL Router pods.
    CODE 
    kubectl delete pod -n mysql -l component=mysqlrouter