Back Up and Restore Using MinIO S3-Compatible Storage

Use MinIO S3-compatible storage to back up and restore Virtual Appliance data.

You can back up Virtual Appliance data to MinIO S3-compatible storage. Complete the following tasks in order:

  1. Set Up MinIO S3-Compatible Storage
  2. Configure MinIO Certificates on the Virtual Appliance
  3. Configure the MinIO Endpoint
  4. Back Up Virtual Appliance Data
  5. Restore Snapshots
  6. Delete Snapshots

Set Up MinIO S3-Compatible Storage

Set up a MinIO server for S3-compatible Virtual Appliance backup storage.

Before you back up Virtual Appliance data using MinIO, set up the MinIO server.

Note: Ensure that the 9000 port is available for the MinIO server.
  1. Download the MinIO server for Linux.
  2. Make the MinIO file executable.
    CODE 
    chmod +x minio
  3. Create a directory to store CA certificates.
    CODE 
    mkdir -p ~/.minio/certs/CAs
cd ~/.minio/certs
  4. Generate a Certificate Signing Request and a private key for the MinIO server.
    CODE 
    openssl genpkey -algorithm RSA -out private.key
openssl req -new -key private.key -out cert.csr
openssl x509 -req -days 365 -in cert.csr -signkey private.key -out public.crt
./minio server --address :9000 /mnt/data

Configure MinIO Certificates on the Virtual Appliance

Configure MinIO CA certificates and Elasticsearch truststore settings for secure snapshot storage.

The Virtual Appliance requires the MinIO CA certificate to establish a secure connection to the MinIO server for snapshot storage.

Copy the MinIO public.crt file to the applicable location on the Virtual Appliance cluster:

PostgreSQL
CODE 
/var/appd/config/postgres-s3-ca.crt
MySQL
CODE 
/var/appd/config/mysql-s3-ca.crt

Because Elasticsearch requires Java KeyStore (JKS) format, convert the public.crt file to a JKS file:

  1. Run the following command to generate the JKS file from the public.crt file:
    CODE 
    keytool -importcert -keystore es-s3-ca.jks -storepass changeit -file public.crt -alias minio
  2. Copy the es-s3-ca.jks file to the following location:
    CODE 
    /var/appd/config
  3. Configure the truststore password in the secrets.yaml file. For example, use changeit. See Edit the secrets.yaml.encrypted file.
    Note:
    • Use the MinIO Console to generate access and secret keys.
    • Set trustStorePassword to the password that you specified for the es-s3-ca.jks file.
    • To synchronize changes in secrets.yaml, run the appdcli sync command.

    Add the following S3 settings:

    CODE 
    # S3 secrets
backup:
  elasticsearch:
    s3:
      access_key: WTFaWld3c1NHWWNOSzBWeDlXaHI=
      secret_key: TWtJY0NweVdIcEhJV0pxdzdIWkRnVld3dzAxODRqbjhnS1pnZHozag==
      trustStorePassword: changeit

Configure the MinIO Endpoint in the globals.yaml.gotmpl File

Configure MinIO repositories and datastore backup settings in the globals.yaml.gotmpl file.

  1. Create a repository to back up data in S3-compatible storage.
    Note: You can create a repository only for Elasticsearch backups. By default, the globals.yaml.gotmpl file uses the repo2 repository for MySQL and PostgreSQL.
    CODE 
    appdcli backup elasticsearch create-s3-repo virtual-appliance-repo-minio
  2. Enable the s3 section for each datastore that you want to back up using the MinIO server. See globals.yaml.gotmpl file.
    Elasticsearch
    JSON 
    # Backup config for datastores
backup:
  elasticsearch:
    s3:
      enabled: true
      endpoint: https://s3-endpoint
      protocol: https
      bucket: es-bucket
      region: us-east-1
      {{ if isFile "/var/appd/config/es-s3-ca.jks" }}
      certFile: {{ readFile "/var/appd/config/es-s3-ca.jks" | b64enc | quote }}
      {{ end }}
    PostgreSQL
    JSON 
    # Backup config for datastores
backup:
  postgresql:
    s3:
      enabled: true
      repoName: repo2
      endpoint: https://10.0.203.142:9000
      bucket: postgresql-bucket
      region: us-east-1
      {{ if isFile "/var/appd/config/postgres-s3-ca.crt" }}
      certFile: {{ readFile "/var/appd/config/postgres-s3-ca.crt" | b64enc | quote }}
      {{ end }}
    MySQL
    JSON 
    # Backup config for datastores
backup:
  mysql:
    s3:
      enabled: true
      repoName: repo2
      endpoint: http://10.0.203.142:9000
      bucket: mysql-bucket
      region: us-east-1
      {{ if isFile "/var/appd/config/mysql-s3-ca.crt" }}
      certFile: {{ readFile "/var/appd/config/mysql-s3-ca.crt" | b64enc | quote }}
      {{ end }}
      # backupType controls which backup method is used when s3 or fs is enabled.
      # xtrabackup:   physical incremental backup via Percona XtraBackup (default)
      # dumpInstance: logical full dump via MySQL Shell
      backupType: xtrabackup
      xtrabackup:
        # Maximum number of full backups to retain in S3/FS (0 = unlimited)
        retentionMaxFull: 3
        # Set to false to skip --lock-ddl (use only if MySQL is quiesced or DDL won't run during backup)
        lockDdl: true
        schedule:
          # Cron expression for full backups (empty string disables)
          full: ""
          # Cron expression for incremental backups (empty string disables)
          incr: ""
          # MySQL pod to backup from
          pod: "appd-mysql-0"
    Note: Do not specify an HTTPS endpoint because MySQL currently supports only an HTTP endpoint.

Back Up Virtual Appliance Data

Back up Virtual Appliance datastore snapshots to an S3 Compatible Storage.

Snapshots capture the state of a datastore and can be used to restore data. You can store multiple snapshots in one repository. The following commands store snapshots in the configured S3-compatible repositories.

Create a snapshot for each datastore:

Elasticsearch
CODE 
appdcli backup elasticsearch create-snapshot <repo-name> <snapshot-name>
Example
CODE 
appdcli backup elasticsearch create-snapshot virtual-appliance-repo-minio es-snapshot
PostgreSQL
CODE 
appdcli backup postgres create-snapshot <repo-name> <snapshot-name>
Example
CODE 
appdcli backup postgres create-snapshot repo2 pg-snapshot
MySQL
Back up the MySQL dumpInstance:
CODE 
appdcli run mysql_backup create-snapshot <snapshot-name>
Back up the MySQL xtrabackup:
  1. Create a full MySQL backup to start a chain.
    CODE 
    # Trigger a full backup
appdcli backup mysql full [--pod appd-mysql-0]

  2. Create any required incremental MySQL backups.

    Use the same MySQL pod for the full backup and every incremental backup in its chain. The default pod is appd-mysql-0.
    CODE 
    appdcli backup mysql incr [--pod <pod>]

Restore Snapshots from the MinIO server

Restore Virtual Appliance datastore snapshots.

Use the snapshots that you have stored in MinIO server to restore the datastores.

Note: You may face issues while restoring a large backups. In such cases, bring the Controller down.
CODE 
helm delete controller -n cisco-controller
  1. List the snapshots in the repository:
    Elasticsearch
    CODE 
    appdcli backup elasticsearch list-snapshots <repo-name>
    CODE 
    appdcli backup elasticsearch restore-snapshot <repository> <snapshot> [<index> ...]
    Example
    CODE 
    appdcli backup elasticsearch list-snapshots virtual-appliance-repo-mino
    PostgreSQL
    CODE 
    appdcli backup postgres list-snapshot <repo-name>
    Example
    CODE 
    appdcli backup postgres list-snapshot repo2
    MySQL
    List the MySQL dumpInstance .
    CODE 
    appdcli backup mysql list

    List the MySQL xtrabackup.

    CODE 
    # List XtraBackup CRs
appdcli backup mysql list

# List all backup chains with size and incremental
appdcli backup mysql list-chains
  2. Restore the datastore by one of the snapshots:
    Elasticsearch
    CODE 
    appdcli backup elasticsearch restore-snapshot <repo-name> <snapshot-name>
    Example
    CODE 
    appdcli backup elasticsearch restore-snapshot virtual-appliance-repo-mino es_snapshot
    PostgreSQL
    CODE 
    appdcli backup postgres restore-snapshot <repo-name>
    Example
    CODE 
    appdcli backup postgres restore-snapshot repo2 pg_snapshot
    MySQL

    Restore the MySQL dumpInstance.

    CODE 
    appdcli backup mysql restore-snapshot <repo-name>
    Restore the MySQL xtrabackup:
    CODE 
    appdcli backup mysql list-chains

# Latest chain and all incremental backups
appdcli backup mysql restore 

# Selected chain and all incremental backups
appdcli backup mysql restore --from-chain <timestamp> 

# Selected chain through an incremental backup, inclusively
appdcli backup mysql restore --from-chain <timestamp> --from-incr <timestamp> 

# Full backup only
appdcli backup mysql restore --from-chain <timestamp> --from-incr full
    Note: You must manually restore the MySQL cluster.

  3. After a MySQL data restore, recover the InnoDB Cluster and verify its status.

    Example:
    CODE 
    appdcli mysql restore
appdcli mysql status --strict

  4. Verify that the restored datastore is healthy and that the expected application data is available.

After the restore is successful, the selected datastore data is restored and validated. If the Controller is down, run the following command:

CODE 
appdcli sync appd <profile>

Delete Snapshots from MinIO S3-Compatible Storage

Delete Virtual Appliance datastore snapshots that you no longer need from MinIO storage.

Delete snapshots that you no longer need from MinIO S3-compatible storage.

Elasticsearch
CODE 
appdcli backup elasticsearch delete-snapshot <repo-name> <snapshot-name>
Example
CODE 
appdcli backup elasticsearch delete-snapshot virtual-appliance-repo-mino es_snapshot
PostgreSQL
CODE 
appdcli backup postgres delete-snapshot <repo-name>
Example
CODE 
appdcli backup postgres delete-snapshot repo2 pg_snapshot
MySQL

Delete the MySQL dumpInstance.

CODE 
appdcli backup mysql delete-snapshot <repo-name>
Delete the MySQL xtrabackup:
CODE 
appdcli backup mysql list-chains
appdcli backup mysql delete-chain <timestamp>

Use --force only when you intentionally delete the latest chain.