Best Practices for Backups

To perform a complete backup of the Controller, the following three directories must be backed up:

  1. Controller install directory.
  2. MySQL datadir.
  3. JRE directory used for Glassfish.

Backing up the entire system each night may not be feasible when dealing with the large amount of data typically generated by a Controller deployment. To balance the risk of data loss against the costs of performing backups, a typical backup strategy calls for backing up the system at different scopes at different times. That is, you may choose to perform partial backups more frequently and full backups less frequently.

The scope of a Controller backup can be categorized into these levels:

  • Level 1: A light backup of the installation environment only.
  • Level 2: A metadata backup involving all metadata associated with the installation except for big data tables.
  • Level 3: Backs up all data, either by performing a cold backup of the /data directory or a hot backup using a third-party tool.

A possible backup strategy may be to perform a level 1 and level 2 backup very frequently, say nightly, and a level 1 and level 3 backup about once a week. In addition to performing a level 1 or 2 backup, you should also back up the data for the Enterprise Console with mysqldump on a regular basis. A level 3 backup also backs up the Enterprise Console data. See Enterprise Console Back Up and Restore for more information.

Light Backup (Level 1)

A light backup targets Controller configuration files like db.cnf and domain.xml. This type of backup lets you avoid having to reconfigure the Controller in case of machine failure.

To perform this type of backup, simply copy everything in the Controller installation directory EXCEPT the data directory.

While it is recommended that you copy the entire Controller home except for the data directory when performing a light backup, particularly before performing a Controller upgrade, there are scenarios in which you may wish to copy only site-specific configuration files. This may be the case if you are migrating an existing Controller configuration to a new Controller installation, for example. For a list of those files, see Migrate the Controller.

Metadata Backup (Level 2)

A metadata backup exports the data that encapsulates the environment monitored by the Controller. Metadata defines the applications monitored by the Controller, business transactions, policies, and so on. It does not include what can be thought of as "runtime data", the big data tables that contain the metrics, snapshots, events, and top summary stats (top SQL, top URLs, and so on) generated in the monitored environment. By backing up metadata, you can avoid having to reconfigure monitored applications in the Controller in the event of a failure.

To perform this type of backup:

  1. Run the script described in Using mysqldump to back up the Controller.
  2. Then augment it with a copy of:
    1. The Controller Java keystore (600 byte file): <controller install>/.appd.scskeystore
    2. And the Enterprise Console keystore: platform-admin/.appd.scs

Complete Backup (Level 3)

A complete backup saves all runtime data associated with the Controller installation. It captures the actual metrics data, snapshots, and so on.

Some third-party backup tools, such as Percona XtraBackup, do not rely on transactions so you can perform a hot backup of your system (that is, back up the Controller database while it is running).

You can perform a complete backup as either:

  1. A cold backup of all three directories (Controller install directory, MySQL datadir, and JRE directory). To perform a cold backup, shut down the Controller app server and database. Then, create an extra copy of the three directories using the cp -r command, the tar utility, rsync, or others.
  2. A hot backup, which means the Controller is running.
    1. If you have a high availability setup for the Controller, you can shut down the database on the secondary Controller. Then, you can perform a cold backup on the secondary Controller and restart the database.

    2. If you do not have a high availability setup for the Controller, use a third-party tool such as Percona XtraBackup to back up the MySQL datadir. Then, use the cp -r command, the tar utility, rsync, or others to back up the Controller install directory and the JRE directory.

      Warning:

      Percona XtraBackup can fail to hot backup Controllers that are too busy. To avoid this error,

      • Back up the secondary Controller instead, if it exists.
      • Increase the MySQL log sizes.
      • Perform the hot backup when the Controller is less busy.