Recreate warm standby after a failover

After a failover, the previous warm standby is now a standalone primary instance of Splunk SOAR (On-premises) and the previous primary is offline or otherwise unavailable. A Splunk SOAR (On-premises) administrator can reconfigure these two instances into a new warm standby pair.

For the rest of this topic the two Splunk SOAR (On-premises) instances will be referred to as either instance A or instance B.

Instance A

The original primary Splunk SOAR (On-premises) instance.

Instance B

The original warm standby instance of Splunk SOAR (On-premises).

Configure instance B as the primary and instance A as the warm standby

This is the easiest way to reconfigure the instances for warm standby after a failover.

The initial states for your instances must be:

  • Instance A, the original primary is online but Splunk SOAR (On-premises) services are not running.
  • Instance B, the former warm standby is now a stand alone Splunk SOAR (On-premises) instance.
CAUTION: If the Splunk SOAR (On-premises) instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

Do these steps as the phantom user.

  1. SSH to instance A. 
     SSH <username>@<instance_A_hostname>
    1. Start PostgreSQL. 
      /<PHANTOM_HOME>/bin/phsvc start postgresql-15
    2. Start pgbouncer. 
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
    3. Turn off primary mode 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
  2. SSH to instance B. 
     SSH <username>@<instance_B_hostname>
    1. Configure instance B as the new primary. 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
  3. SSH to instance A. 
     SSH <username>@<instance_A_hostname>
    1. Configure instance A as the new warm standby. 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Now the two instances are configured for warm standby. Instance B is now the primary and instance A is now the warm standby.

Configure instance A as the primary and instance B as the warm standby

This option returns the instances to the same roles they served before the failover. This can be done after you have configured the instance B as the primary using the steps in the earlier section.

CAUTION: Each time warm standby is configured the database on the standby instance is erased and the entire Splunk SOAR (On-premises) PostgreSQL database has to be streamed from the primary.

The initial states for your instances must be:

  • Instance B, the former warm standby is now a stand alone Splunk SOAR (On-premises) instance. All Splunk SOAR (On-premises) services are running.
  • Instance A, the original primary is configured as the warm standby. All Splunk SOAR (On-premises) services are running.
CAUTION: If the Splunk SOAR (On-premises) instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.
  1. SSH to instance B. 
     SSH <username>@<instance_B_hostname>
    1. Stop Splunk SOAR (On-premises) services. 
      /<PHANTOM_HOME>/bin/stop_phantom.sh
  2. SSH to instance A. 
     SSH <username>@<instance_A_hostname>
    1. Configure instance A as the primary. 
       phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --convert-to-primary
      Warm standby is disabled. Instance A is a standalone Splunk SOAR (On-premises) instance, while instance B is idle and all Splunk SOAR (On-premises) services have been shut down.
      CAUTION: If the Splunk SOAR (On-premises) instances are not in the described states, stop. Check for and do any steps which have been missed before proceeding.
  3. SSH to instance B. 
     SSH <username>@<instance_B_hostname>
    1. Start PostgreSQL. 
      /<PHANTOM_HOME>/bin/phsvc start postgresql-15
    2. Start pgbouncer. 
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
    3. Turn off primary mode 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
  4. SSH to instance A. 
     SSH <username>@<instance_A_hostname>
    1. Configure instance A as primary. 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
  5. SSH to instance B. 
     SSH <username>@<instance_B_hostname>
    1. Configure instance B as the warm standby. 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Instance A and B are configured as a warm standby pair. Instance A is the primary, and instance B is the warm standby.