Backup and restore using MinIO S3 compatible storage

Follow the steps to back up and restore datastores using MinIO S3 storage:

Back Up Elasticsearch to MinIO S3-Compatible Storage

Create and verify an Elasticsearch snapshot in a MinIO S3-compatible repository.

  1. Create the S3 snapshot repository.
    CODE 
    appdcli backup elasticsearch create-s3-repo repo2
  2. Verify that repo2 is listed as an s3 repository with the expected bucket, endpoint, protocol, and region.
    CODE 
    appdcli backup elasticsearch list-repos
  3. Create a snapshot.
    CODE 
    appdcli backup elasticsearch create-snapshot repo2 <snapshot-name>
  4. Wait until the snapshot state is SUCCESS.
    CODE 
    appdcli backup elasticsearch list-snapshots repo2
  5. Optional: Verify that snapshot objects are present in the MinIO bucket.
    CODE 
    kubectl exec -n minio minio-0 -- mc ls local/es-bucket --recursive --insecure
The Elasticsearch snapshot is stored in MinIO S3-compatible storage.

Restore Elasticsearch from MinIO S3-Compatible Storage

Restore an Elasticsearch snapshot in place from MinIO S3-compatible storage.

The restore command closes existing indexes before it restores them. You do not need to scale down the events service or Elasticsearch for an in-place restore.
  1. Restore the selected snapshot.
    CODE 
    appdcli backup elasticsearch restore-snapshot repo2 <snapshot-name>

    Wait for the command to report that the restore completed.

  2. Verify Elasticsearch cluster health.
    CODE 
    kubectl get elasticsearch -n es

    Verify that health is green and phase is Ready.

Recover Elasticsearch from MinIO S3-Compatible Storage After PVC Loss

Rebuild Elasticsearch with new data PVCs and restore a MinIO S3-compatible snapshot.

Verify the availability of the selected snapshot and MinIO bucket.
CAUTION:

  • This procedure stops the events service and Elasticsearch.

  • Deleting the Elasticsearch data PVCs permanently removes the data stored on them. Verify the PVC names before deletion.

  1. Stop the events service and Elasticsearch.
    CODE 
    kubectl scale deploy events-ss -n cisco-events --replicas=0
kubectl annotate elasticsearch appd -n es "elasticsearch.k8s.elastic.co/cluster-uuid-"
kubectl scale statefulset appd-es-node -n es --replicas=0
kubectl scale statefulset es-shell -n es --replicas=0
  2. List and delete the lost or corrupted Elasticsearch data PVCs.
    CODE 
    kubectl get pvc -n es
kubectl delete pvc <elasticsearch-data-pvc-0> <elasticsearch-data-pvc-1> <elasticsearch-data-pvc-2> -n es
kubectl get pvc -n es
  3. Start Elasticsearch and wait for the empty cluster to become ready.
    CODE 
    kubectl scale statefulset appd-es-node -n es --replicas=3
kubectl scale statefulset es-shell -n es --replicas=3
kubectl get elasticsearch -n es
  4. Re-create the repository registration and restore the snapshot.
    CODE 
    appdcli backup elasticsearch create-s3-repo repo2
appdcli backup elasticsearch restore-snapshot repo2 <snapshot-name>
  5. Start the events service.
    CODE 
    kubectl scale deploy events-ss -n cisco-events --replicas=3

Back Up PostgreSQL to MinIO S3-Compatible Storage

Create and verify a PostgreSQL backup in the S3-backed pgBackRest repository.

Configure repo2 as the S3-backed pgBackRest repository.
Ensure that the S3 backup option is enabled in globals.yaml.gotmpl.
CODE 
s3:
      enabled: true
  1. Create a backup.
    CODE 
    appdcli backup postgres create-snapshot repo2

    Record the generated perconapgbackup resource name.

  2. Wait until the backup status is Succeeded.
    CODE 
    kubectl get perconapgbackup <backup-resource-name> -n postgres
  3. List the backups recorded in the repository.
    CODE 
    appdcli backup postgres list-snapshots repo2
  4. Optional: Verify that backup objects are present in the MinIO bucket.
    CODE 
    kubectl exec -n minio minio-0 -- mc ls local/postgres-backup --recursive --insecure

Restore PostgreSQL from MinIO S3-Compatible Storage

Restore the latest PostgreSQL backup in place from the S3-backed pgBackRest repository.

The command restores one instance and automatically recycles replica pods and PVCs that have a PostgreSQL system-ID mismatch.
  1. List available backups and review the RESTORE column.
    CODE 
    appdcli backup postgres list-snapshots repo2

    If the repository lineage has changed, use --set=<label> with the restore command.

  2. Restore the backup and confirm the prompt.
    CODE 
    appdcli backup postgres restore-snapshot repo2
    Use the --force option only when you intend to skip the confirmation prompt.
    Warning:

    If the backup version does not match the version on the replica pods, the pods must be restarted. This utility automatically restarts the affected replica pods and deletes their PVCs.

    If you understand and accept this action, type yes to continue.

  3. Wait until the restore status is Succeeded.
    CODE 
    kubectl get perconapgrestore -n postgres
  4. Verify that all PostgreSQL instances are ready.
    CODE 
    kubectl get perconapgcluster -n postgres

Recover PostgreSQL from MinIO S3-Compatible Storage After PVC Loss

Rebuild PostgreSQL instance PVCs and restore a labeled backup from MinIO S3-compatible storage.

Record the label of a backup taken before the PVC loss.
CAUTION:

Pausing the cluster stops PostgreSQL.

Delete only PostgreSQL instance data PVCs. Do not delete a backup repository PVC.

  1. Pause the cluster and wait until all instance pods terminate.
    JSON 
    kubectl patch perconapgcluster appd-postgres -n postgres --type=merge -p '{"spec":{"pause":true}}'
kubectl get pods -n postgres -l postgres-operator.crunchydata.com/instance-set=instance
  2. List and delete the lost or corrupted instance PVCs.
    CODE 
    kubectl get pvc -n postgres
kubectl delete pvc <postgres-instance-pvc-1> <postgres-instance-pvc-2> <postgres-instance-pvc-3> -n postgres
  3. Unpause the cluster and wait for new PVCs and pods.
    JSON 
    kubectl patch perconapgcluster appd-postgres -n postgres --type=merge -p '{"spec":{"pause":false}}'
kubectl get pods -n postgres -w
  4. List backups and identify the pre-loss label.
    CODE 
    appdcli backup postgres list-snapshots repo2
  5. Restore the labeled backup.
    CODE 
    appdcli backup postgres restore-snapshot repo2 --set=<backup-label>
  6. Verify the restore and replication health.
    CODE 
    kubectl get perconapgrestore -n postgres
kubectl exec -n postgres <primary-pod> -c database -- patronictl list

    Verify a running leader and streaming replicas.

Back Up MySQL to MinIO S3-Compatible Storage

Create a full MySQL XtraBackup chain in MinIO S3-compatible storage.

Configure MySQL XtraBackup to use the S3 backend.
  1. Create a full backup to start a new chain.
    CODE 
    appdcli backup mysql full

    Use the same MySQL pod for a full backup and all incremental backups in its chain.

  2. Wait until the XtraBackup phase is Completed.
    CODE 
    appdcli backup mysql list
  3. List backup chains and review the generated listing job.
    CODE 
    appdcli backup mysql list-chains
kubectl logs -n mysql -l job-name=<xtrabackup-listing-job>
  4. Optional: Verify that backup objects are present in the MinIO bucket.
    CODE 
    kubectl exec -n minio minio-0 -- mc ls local/mysql-backup/xtrabackup/ --insecure

Restore MySQL from MinIO S3-Compatible Storage

Stop the MySQL cluster, restore an XtraBackup chain from MinIO S3-compatible storage, and rebuild the InnoDB Cluster.

List backup chains and record the chain timestamp to restore.
Warning: This procedure causes a full MySQL outage. Run the commands in order and wait for each stated condition.
  1. Reduce the cluster to one instance, stop the operator, and stop the StatefulSet.
    JSON 
    kubectl patch innodbcluster appd-mysql -n mysql --type=merge -p '{"spec":{"instances":1}}'
kubectl scale deploy mysql-operator -n mysql-operator --replicas=0
kubectl scale sts appd-mysql -n mysql --replicas=0

    Wait until the MySQL pods show Completed.

  2. Restore the selected backup chain and confirm the prompt.
    CODE 
    appdcli backup mysql restore --from-chain <chain-timestamp>
  3. Wait for the XtraRestore phase to show Completed.
    CODE 
    kubectl get xtrarestore <xtrarestore-name> -n mysql
  4. Clear the first pod's finalizer and start one MySQL instance while the operator remains stopped.
    JSON 
    kubectl patch pod appd-mysql-0 -n mysql -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl scale sts appd-mysql -n mysql --replicas=1
  5. Reset the metadata schema and create a single-node InnoDB Cluster.
    CODE 
    kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh --sql -u root -p'<root-password>' -e "SET GLOBAL super_read_only=OFF; SET GLOBAL read_only=OFF; DROP DATABASE IF EXISTS mysql_innodb_cluster_metadata;"
kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh -u root -p'<root-password>' -- dba create-cluster appd-mysql --force
  6. Start the operator and return the cluster to three instances.
    JSON 
    kubectl scale deploy mysql-operator -n mysql-operator --replicas=1
kubectl patch innodbcluster appd-mysql -n mysql --type=merge -p '{"spec":{"instances":3}}'
kubectl get innodbcluster appd-mysql -n mysql

    Wait for the status to show ONLINE with three online instances.

  7. Recycle the MySQL Router pods.
    CODE 
    kubectl delete pod -n mysql -l component=mysqlrouter

Recover MySQL Replicas After a MinIO S3-Compatible Restore

Join MySQL replicas manually when PVC loss prevents automatic recovery after an S3 restore.

Use this procedure when the restored cluster does not reach ONLINE and an affected replica reports Shell Error (51113): Metadata Schema not found.
  1. Delete the affected replica pod and its data PVC.
    CAUTION: Deleting the PVC permanently removes that replica's data. Verify the health of the restored primary first.
    CODE 
    kubectl delete pod <mysql-replica-pod> -n mysql --force --grace-period=0
kubectl delete pvc <mysql-replica-pvc> -n mysql
  2. Wait for the StatefulSet to create a new pod and PVC.
    CODE 
    kubectl get pods -n mysql -l component=mysqld
  3. From appd-mysql-0, add the instance with clone-based recovery.
    JSON 
    kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh --py -u root -p'<root-password>' -e "
cluster = dba.get_cluster('appd-mysql')
cluster.add_instance({'host': '<replica-host>', 'port': 3306, 'user': 'root', 'password': '<root-password>'}, {'recoveryMethod': 'clone'})
"
  4. Repeat the recovery for each affected replica and verify an ONLINE state for every member.
    CODE 
    kubectl exec -n mysql appd-mysql-0 -c mysql -- mysqlsh --sql -u root -p'<root-password>' -e "SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;"
  5. Recycle the MySQL Router pods.
    CODE 
    kubectl delete pod -n mysql -l component=mysqlrouter