Initiate Controller Database Incremental Replication

Re-enable Broken Replication

Incremental replication, replication via rsync when the primary database is up, is required in cases where the database replication on the secondary Controller is lagging behind the primary Controller by more than three days. This type of replication allows the primary Controller to keep operating while the disk contents are copied to the secondary node.

To initiate incremental replication:

  1. Run the following command on the Enterprise Console host:This launches a continuously running background job. 
    bin/platform-admin.sh submit-job --service controller --job incremental-replication
    This launches a continuously running background job.

  2. Make sure replication occurs four or more times by running either one of the following commands:

    1. CODE 
      cd <controller_home>/controller-ha
./ha_replicate.sh -r status
    2. CODE 
      cd <controller_home>/controller-ha/tmp
cat replication.status
    Note: If replication fails, go to the secondary host and stop all rsync and ha-replicate.sh processes. Then try running the incremental-replication job again.
  3. Finalize the job by running the following command on the Enterprise Console host:This stops the incremental replication loop. The command will restart the primary Controller, resulting in downtime. 
    bin/platform-admin.sh submit-job --service controller --job finalize-replication
    This stops the incremental replication loop. The command will restart the primary Controller, resulting in downtime.
  4. Make sure replication is working by checking that there is no significant gap between the primary and secondary Controllers. You can run the following command on the Enterprise Console host to check the replication status:It may take a few minutes to display the secondary status. 
    bin/platform-admin.sh show-service-status --platform-name <platform_name> --service controller
    It may take a few minutes to display the secondary status.

Add a Secondary Controller Using Incremental Replication

You can convert a single Controller with a large amount of data to an HA pair by using incremental replication. This way, you can rsync most of the Controller data while the Controller is still running, limiting the downtime of adding a secondary Controller.

To add a secondary Controller using incremental replication:

  1. Start the incremental replication, giving host and rsync parameters: 
    bin/platform-admin.sh submit-job --service controller --job incremental-replication --args controllerSecondaryHost=1.1.1.1 rsyncThrottle=40000 rsyncCompress=true
    This launches a continuously running background job.
  2. Make sure replication occurs four or more times, by checking <controller_home>/controller-ha/tmp/replication.status on the primary database host.Sample rsync status file output: 
    rsync started at Mon Mar  5 11:49:56 PST 2018
rsync completed at Mon Mar  5 11:50:56 PST 2018
rsync started at Mon Mar  5 11:51:01 PST 2018
rsync completed at Mon Mar  5 11:51:11 PST 2018
    Note: If replication fails, go to the secondary host and stop all rsync and ha-replicate.sh processes. Then try running the incremental-replication job again.
  3. Run the add secondary job. The Enterprise Console will perform a final rsync and add the secondary job. 
    bin/platform-admin.sh submit-job --service controller --job add-secondary --args controllerSecondaryHost=secondary mysqlRootPassword=‘password'
    The command will restart the primary Controller, resulting in downtime.
    Note: Until you trigger the add-secondary command, the secondary Controller is not added to the Enterprise Console platform. Therefore, the Enterprise Console will not be able to perform any other operations on the secondary Controller.

If you need to stop replication, you can run the following command:

bin/platform-admin.sh submit-job --service controller --job stop-incremental-replication

Set Replication Factors for Rsync Threads

Using the Enterprise Console UI or the CLI, you can set the number of parallel rsync threads as a job parameter when you perform incremental or finalize replication.

  • From the Enterprise Console UI:

    1. Log in to the Enterprise Console and access the Controller page.

    2. From the More menu, based on which replication you are performing, select either Incremental Replication or Finalize Replication.Increment or Finalize Replication

    3. Enter a number in the Number of parallel rsync threads field and click Submit. The default value is 1.Number of parallel rsync threads

  • From the CLI, based on which replication you are performing, run either of the following commands from the Enterprise Console host and set the numberThreadForRsync argument.

    CODE 
    bin/platform-admin.sh submit-job --job incremental-replication --args numberThreadForRsync=<number> bin/platform-admin.sh submit-job --job finalize-replication --args numberThreadForRsync=<number>

Enable MySQL Parallel Replication

Using the Enterprise Console UI or the CLI, you can enable MySQL (available from MySQL 5.7) parallel replication when you perform finalize replication.

  • From the Enterprise Console UI:

    1. Log in to the Enterprise Console and access the Controller page.

    2. From the More menu, select Finalize Replication.Finalize Replication

    3. Select the Database parallel replication check box to enable parallel replication with the MySQL database.Database parallel replication checkbox

    4. Click Submit.

  • From the CLI, run the following command from the Enterprise Console host to enable MySQL parallel replication. The default value is true.

    CODE 
    bin/platform-admin.sh submit-job --job finalize-replication --args dbParallelReplication=true

Troubleshooting the Incremental Replication Status

If your first incremental replication run is taking longer than usual, you can check the replication status by executing either one of the below commands:

  • CODE 
    cd <controller_home>/controller-ha
./ha_replicate.sh -r status
  • CODE 
    cd <controller_home>/controller-ha/tmp
cat replication.status