Upgrade the Virtual Appliance in KVM

Upgrade the Virtual Appliance in KVM by deploying new virtual machines and restoring backed-up data.

Upgrading the Virtual Appliance involves:

  • Backing up the hard disks of the existing virtual machines.
  • Deploying new virtual machines by using QCOW2 image.
  • Restoring the backup to their hard disks.
  • Attaching the new hard disks to the older virtual machines

Splunk AppDynamics On-Premises Virtual Appliance provides the reference script that helps in upgrading the Virtual Appliance.

Standard Deployment
Sl No Steps
1 Back up the Virtual Appliance Data
2 Upgrade the Virtual Appliance
3 Verify the Deployment Status
4 Restore Data in the Virtual Appliance
Hybrid Deployment
Sl No. Steps
1 Back up the Virtual Appliance Data
2 Upgrade the Virtual Appliance
3 Restart the Standalone Controller
4 Restart the Standalone Controller
5 Generate the Hybrid Configuration File
6 Configure the Standalone Controller for Hybrid Connectivity
7 Restart the Standalone Controller

Standard Deployment

Follow the steps to upgrade the Virtual Appliance in KVM.

Back up the Virtual Appliance Data

Back up Virtual Appliance data before upgrading virtual machines in KVM.

Before you begin the upgrade, complete the following steps:

  1. Download the Virtual Appliance upgrade QCOW2 file.
    1. Log in to the Downloads portal using your credentials.
    2. In Downloads, select the Virtual Appliance tab.
    3. Select QCOW2 from the Type list.
    4. Select Download.
    Note: The file available in the downloads portal is used for both installation and upgrade procedures.
  2. Shutdown the Secure Application, Anomaly Detection, and remaining services. Then, shutdown Splunk AppDynamics.
    CODE 
    appdcli stop secureapp
appdcli stop aiops
appdcli stop [services]
appdcli stop appd
<<Wait for pods to terminate before deleting operators>>
appdcli stop operators
    Note:

    • Ensure that there are no pods in the running state before you start the data backup.

    • Sometimes, IOException error occurs in the Controller UI. See Troubleshoot Virtual Appliance Issues
    .
  3. Optional: Create a migration script for custom actions. See Migrate Custom Actions.
  4. Generate a backup file.

Generate a Backup File

Generate a backup file and save it outside the Virtual Appliance cluster.

Generate a backup file and save it outside the Virtual Appliance cluster.
  1. Run the backup command:
    CODE 
    appdcli platform backup
  2. Copy the generated <backup_tar> file outside the cluster. If you have custom certificates, copy them to the same external location.

Upgrade the Virtual Appliance

Upgrade the Virtual Appliance by running the upgrade script with the QCOW2 file and configuration file.

  1. Download the upgrade script from the Splunk AppDynamics GitHub repository.
  2. Run the following command by specifying the locations  of the QCOW2 file and config.cfg .
    CODE 
    ./upgrade-cluster ../<filename>.qcow2 config.cfg

This script verifies the virsh

  1. Stops the virtual machine.

  2. Uploads the new QCOW2 image to the node storage pool.

  3. Deletes the OS disk image. However, it retains the data disk image.

  4. Creates a new OS disk from the template image.

  5. Starts the virtual machine.

Verify the Deployment Status

Verify the deployment status of Virtual Appliance virtual machines and services in VMware Esxi.

Verify the deployment of virtual machines:

  1. Verify whether the deployment of virtual machines is successful:
    CODE 
    appdctl show boot
    Note: Ensure that the services in each node is active. If any service appears as Failed, restart that virtual machine. You might have to redeploy the virtual machine if it is still failing.
  2. Create a three-node cluster:
    CODE 
    appdctl cluster init <Node-2-IP> <Node-3-IP>
  3. Verify node and MicroK8s status:
    CODE 
    appdctl show cluster
microk8s status
    Confirm that RUNNING is true for all three voter nodes.
    Note: You must re-login to the terminal if the following error appears:
    CODE 
    Insufficient Permissions to Access Microk8s

Restore Data in the Virtual Appliance

Restore backed-up data to the Virtual Appliance after completing a VMware ESXi upgrade.

  1. Copy the previously generated <backup_tar> file to one cluster node. See Prerequisites.
  2. Restore the persistent-volume specifications.
    CODE 
    appdcli platform restore <backup_tar>
  3. Respond to the prompt for self-signed ingress CA certificates:
    • Choose 0 if certificates are configured for DNS.
    • Choose 1 to restore and configure the self-signed certificates.
  4. Verify that all PVCs are in the Bound state:
    CODE 
    kubectl get pvc -A
  5. Start the following services and operators:
    CODE 
    appdcli start appd [Profile]
appdcli start aiops [Profile]
appdcli start secureapp [Profile]
appdcli start [Services] [Profile]
    Note: Sometimes, the MySQL Router pods might fail to start. See MySQL Router Pods Fail to Start.
  6. Sync to running controller.
    CODE 
    appdcli controller actions-sync
After the upgrade, the cluster uses the existing data disk and becomes functional.

Generate the Hybrid Configuration File

Generate the hybrid configuration file when custom certificates are not used for Ingress and Kafka clusters.

Hybrid deployments only: Generate this file when custom certificates are not used for the Ingress and Kafka clusters

Ensure that you have the latest CA certificates obtained after installing services. If not, update the CA certificates and regenerate the hybrid configuration file after restarting the service.

  1. On the Virtual Appliance, generate the hybrid configuration:
    CODE 
    cd ~/appd-charts/utils
bash prepare-hybrid.sh

    Sample output:

    CODE 
    Writing Kafka CA Cert
Writing Schema registry CA Cert
Writing AnomalyDetectionKafkaSSLProducer
Writing AnomalyDetectionKafkaConsumerSSL
Writing configure script
Creating archive of hybrid config
hybrid-config/
hybrid-config/kafka-ca.crt
hybrid-config/AnomalyDetectionKafkaConsumerSSL
hybrid-config/configure.sh
hybrid-config/schema-registry-ca.crt
hybrid-config/AnomalyDetectionKafkaSSLProducer
Completed

    The script creates hybrid-config.tar, containing Kafka and Schema Registry CA certificates, producer and consumer SSL configuration, and configure.sh.

  2. Copy the hybrid-config.tar file to the standalone Controller.
  3. Extract and run the configuration script:
    CODE 
    tar -xvf hybrid-config.tar
  4. Change to the extracted configuration directory and run the script:
    CODE 
    cd ~/hybrid-config
bash configure.sh
  5. Enter the actual Controller home when prompted:
    CODE 
    Controller home [/opt/appdynamics/platform/product/controller]: /path/to/platform/controller
    Note:

    Virtual Appliance certificates expire after five years. Regenerate the hybrid configuration and copy updated certificates to the Classic Controller as part of certificate rotation.

    For more information on how to audit and rotate certificates in Virtual Appliance, see Manage Virtual Appliance Certificates.

  6. Restart the classic Controller.

Configure the Standalone Controller for Hybrid Connectivity

Complete these checks before running the script:

  • Extract the generated hybrid-config archive on the Controller host.
  • Run the script from inside the extracted hybrid-config directory.

  • Identify the actual Controller home: The directory containing db/, tools/, and appserver/.

    Tip: /home/appdynamics/appdynamics/platform/controller. The suggested default is /opt/appdynamics/platform/product/controller, but the installation path can vary.
  • Obtain the DNS name configured as hybrid.mysql.dbHost in the Virtual Appliance.
  • Use an account with permission to update the Controller and MySQL certificate files.

Controller home example: /home/appdynamics/appdynamics/platform/controller. The suggested default is /opt/appdynamics/platform/product/controller, but the installation path can vary.

  1. Change to the extracted configuration directory and run the script:
    CODE 
    cd ~/hybrid-config
bash configure.sh
  2. Enter the actual Controller home when prompted:
    CODE 
    Controller home [/opt/appdynamics/platform/product/controller]: /path/to/platform/controller
    Note: If the Controller home is rejected, verify that the directory contains db/db.cnf.
  3. Review the proposed changes, then enter y to continue:
    CODE 
    This operation will:
  - Replace the Kafka and Schema Registry client truststores
  - Update the Controller CA truststore
  - Write obfuscated Kafka producer and consumer configuration
  - Replace the MySQL server certificate after creating a backup

Continue? [y/N]: y
  4. Confirm the reachable Controller address.
    The script detects a local IPv4 address. Confirm that the Virtual Appliance cluster can reach this address. Enter n to provide a different address.
    PYTHON 
    Detected Controller IP: 192.0.2.10
Is this address reachable from the Virtual Appliance cluster? [Y/n]: y
  5. Provide the hybrid database DNS name.
    Enter the exact hostname configured as hybrid.dbHost. The script adds this value to the MySQL certificate Subject Alternative Name (SAN).
    CODE 
    DNS name [192.0.2.10.nip.io]: controller.example.com

Keystore password behavior: The script first tries the standard keystore password, changeit. If that password cannot open the existing Controller CA truststore, the script prompts securely for the configured password. Typed password characters are not displayed.

The selected password is used consistently for:

  • The Kafka client truststore.
  • The Schema Registry client truststore.
  • The existing Controller CA truststore.
  • The truststore-password fields in the obfuscated producer and consumer configurations.
Note: If the keystore password is rejected, provide the password currently configured for appserver/jetty/etc/cacerts.jks.

Changes made by the script:

Item Location relative to Controller home Action
Kafka client truststore pi-kafka-ssl-config/kafka.client.truststore.jks Created or replaced using kafka-ca.crt.
Schema Registry client truststore pi-kafka-ssl-config/schema-registry.client.truststore.jks Created or replaced using schema-registry-ca.crt.
Controller CA truststore appserver/jetty/etc/cacerts.jks Replaces the k8s-cluster alias with schema-registry-ca.crt.
Kafka client configurations pi-kafka-ssl-config/AnomalyDetectionKafka* Writes obfuscated producer and consumer configuration.
MySQL server certificate Database data directory from db/db.cnf Generates a 365-day certificate containing the supplied DNS SAN.
Before replacement, the script creates timestamped backups of the Controller CA truststore and the existing MySQL server certificate. The final summary displays the complete backup paths.
Note: Use the timestamped backup paths printed by the script when following an approved rollback procedure.

If the configuration is successful, the following message will be displayed:

CODE 
Hybrid Controller configuration completed successfully.
Important:

After the script completes successfully, restart the Controller and its MySQL service. The Controller truststore and MySQL certificate changes do not take effect until the affected services are restarted.

Restart the Standalone Controller

Restart the Standalone Controller and MySQL service after completing configuration changes.

  • Run the following command to restart the standalone Controller:
    CODE 
    cd ~/appdynamics/platform/product/controller/bin/
./controller.sh stop
./controller.sh start

    This command also restarts the MySQL service.

    Alternatively, you can restart the controller from Enterprise Console. See Start or Stop the Controller.

  • For high-availability scenario, you must use Enterprise Console to restart the controller. See Start or Stop the Controller.