Set Up a High Availability Deployment

This page describes how to set up and deploy Controllers as a high availability (HA) pair. For installation and upgrade details, see Custom InstallNo Content found for /db/organizations/splunk/repositories/appdonprem-26_6_0/AppDynamics/documents/content/on_premises/c_custom_install.dita and Upgrade an HA Pair.

Warning: The Enterprise Console HA deployment works on Linux systems only. Controller HA pairs are not available on Windows machines using Enterprise Console.

About the HA Deployment Using the Enterprise Console

The Enterprise Console automates HA-related setup and administration tasks for the Linux operating system. It does not require sudo privileges and can be deployed as a non-root user on Unix operating systems. It works with most flavors of Linux, including Ubuntu and Red Hat/CentOS.

Warning: The servers of Controllers in an HA pair must be identical in terms of OS, CPU, RAM, and Disk. See

Controller System Requirements

.

You can:

  • Configure Controllers in a high availability pair arrangement.
  • Use the Enterprise Console to monitor the health of the primary Controller, App Server, and database, and failover to the secondary when needed.
  • Use scripts that allows you to install the Controllers as a Linux service, and gracefully stop and start service in the event of a machine reboot.
  • Failover to a secondary Controller manually (for example, when you need to perform maintenance on the primary).
  • Revive a Controller (restore a Controller as an HA secondary after its database is more than seven days behind the primary as a replica).
  • Set up a Controller HA pair.

Deploying Controllers as an HA pair ensures that service downtime in the event of a Controller machine failure is minimized. It also facilitates other administrative tasks, such as backing up data. For more background information, including the benefits of HA, see Controller High Availability (HA).

User Privilege Escalation Requirements

After installing a Controller high availability via the Enterprise Console, it is recommended to install MySQL as a Linux service. This is to prevent against MySQL data integrity issues. Installing the Controller and MySQL as a Unix service will ensure that whenever the machine reboots, the service will be shut down and started gracefully.

  • /etc/sudoers.d/appdynamics contains entries to allow the Splunk AppDynamics user to access the /sbin/service utility using sudo without a password. This mechanism is not available if theSplunk AppDynamics user is authenticated by LDAP.

  • /sbin/appdservice is a setuid root program distributed in source form in <controller_home>/controller-ha/init/appdservice.c. It is written explicitly to support auditing by security audit systems. The install-init.sh script compiles and installs the program. It is executable only by the Splunk AppDynamics user and the root user. The script requires a C compiler to be available on the system. You can install a C compiler using the package manager for your operating system. For example, on Yum-based Linux distributions, you can use the following command to install the GNU Compiler, which includes a C compiler:

    sudo yum install gcc

Set Up the Controller High Availability Pair

To set up high availability:

Step 1: Configure the Controller High Availability Pair Environment

The following sections provide more information on how to configure a few of the system requirements. They describe how to configure the settings on Red Hat Linux as a sample deployment. Note that the specific steps for configuring these requirements may differ on different systems. Consult your system documentation for specific details.

Host Reverse Lookups

You need to set up a reliable symmetrical reverse host lookup on each machine. To do this, enter the hostnames of the pair into the hosts files (/etc/hosts) on each machine. This is preferable over other approaches, such as using reverse DNS, which adds a point of failure.

To enable reverse host lookups, on each host:

  1. In /etc/nsswitch.conf , enter files before dns to have the hosts file entries take precedence over DNS. For example: hosts: files dns

  2. In /etc/hosts file, add an entry for each host in the HA pair. For example:192.168.144.128 host1.domain.com host1192.168.144.137 host2.domain.com host2
    Note: To reduce errors, use the correct format of /etc/hosts files. If you have both dotted hostnames and short versions, you need to list the dotted hostnames with the most dots first and the other versions subsequently. This should be done consistently for both HA server entries in each of the two /etc/hosts files. Note that in the examples provided, the aliases are listed last.

Set Up the SSH Key

SSH must be installed on both hosts in a way that gives the user who runs the Controller passwordless SSH access to the other Controller system in the HA pair. You can accomplish this by generating a key pair on each node, and placing the public key of the other Controller into the authorized keys (authorized_keys) file on each Controller.

The following steps describe how to perform this configuration. The instructions assume an Splunk AppDynamics user named appduser, and the Controller hostnames are node1, the active primary, and node2, the secondary. Adjust the instructions for your particular environment. Also note that you may not need to perform every step (for example, you may already have the .ssh directory and don't need to create a new one).

Although not shown here, some of the steps may prompt you for a password.

On the primary (node1) host:
  1. Change to the Splunk AppDynamics user, appduser in our example: 
    su - appduser
  2. Create a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows: 
    mkdir -p .ssh
chmod 700 .ssh
  3. Generate the RSA-formatted key: 
    ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem
  4. Secure copy the key to the other Controller: 
    scp .ssh/id_rsa.pub node2:/tmp
On the secondary (node2) host:
  1. As you did for node1, run these commands: 
    su - appduser
mkdir -p .ssh
chmod 700 .ssh
ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem
scp .ssh/id_rsa.pub node1:/tmp
  2. Add the public key of node1 that you previously copied to the secondary Controller host's authorized keys and set permissions on the authorized keys file: 
    cat /tmp/id_rsa.pub >> .ssh/authorized_keys
chmod 700 ~/.ssh/authorized_keys
On the primary (node1) again:
  1. Move the secondary's public key to the authorized keys 
    cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh/authorized_keys

To test the configuration, enter:

ssh -oNumberOfPasswordPrompts=0 <other_node> "echo success"

Verify that the echo command is successful.

Step 3: Activate the Controller High Availability Pair

This step ensures that the Enterprise Console is no longer in the critical path of the HA Controller failover process.

Open a command shell on the Enterprise Console host and enter:

CODE 
platform-admin.sh submit-job --service controller  --job activate-ha-modules

Your output should be similar to the following:

Output

Convert a Standalone Controller to a Controller High Availability Pair

You can convert a standalone Controller to a Controller HA pair through the Enterprise Console GUI. Ensure that you have completed the prerequisites in the above Configure the Controller High Availability Pair Environment section that requires both the primary and secondary hosts to talk to each other via passwordless SSH.

Additionally, you can use incremental replication to add a secondary Controller. See Initiate Controller Database Incremental Replication for more information.

If you are starting from a fresh installation, you will need to first create a platform, then add two credentials and hosts for your HA pair.

To convert a standalone Controller to a Controller HA pair:

  1. Open the Enterprise Console GUI.
  2. Verify that the credentials and hosts you want to use are added to the Splunk AppDynamics On-Premises platform. For more information, see Administer the Enterprise ConsoleNo Content found for /db/organizations/splunk/repositories/appdonprem-26_6_0/AppDynamics/documents/content/on_premises/c_administer_the_enterprise_console.dita.
    1. On the Credential page, add the SSH credentials for the host you want to install the secondary Controller on. You can also run the following command on the Enterprise Console host: 
      bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
      Note: Remember to provide the private key file for the Enterprise Console machine when adding a credential.
    2. On the Hosts page, add the host. You can also run the following command on the Enterprise Console host: 
      bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
      The Enterprise Console uses this host for the HA pair.
  3. On the Controller page, click Add Secondary Controller, and complete the wizard:
    1. Select the Controller Secondary Host that you added for the secondary Controller.
    2. Optional:Enter the External URL. This is the external load balancer URL, which should reflect this format: http(s)://<external.vip>:<port>
    3. Enter the DB Root Password, and re-enter it for confirmation.
      Warning: Ensure to provide the same passwords during the secondary server installation as those that you provided for the primary server.
  4. Select Submit.
Your HA pair will automatically set up, each with their own MySQL node.