Upgrade the Virtual Appliance in VMware

Upgrade the Virtual Appliance in VMware ESXi by deploying new virtual machines from an OVA file.

Upgrade the Virtual Appliance in VMware ESXi by deploying new virtual machines from an OVA file. This process involves:

  • Backing up the existing Virtual Appliance data.
  • Deploying new virtual machines by using OVA image.
  • Removing the new data disks and attaching the preserved data disks from the previous virtual machines.
  • Restoring the backup and starting the services.

Splunk AppDynamics On-Premises Virtual Appliance provides the reference script that helps in upgrading the Virtual Appliance.

Note: This document contains links to VMware documentation. Splunk AppDynamics makes no representation as to the accuracy of VMware documentation because VMware controls its own documentation.
Standard Deployment
Sl No Steps
1 Back Up the Virtual Appliance Data
2 Prepare to Upgrade the Virtual Appliance
3 Upgrade with the OVA File
4 Verify the Deployment Status
5 Restore Data in the Virtual Appliance
Hybrid Deployment
Sl No. Steps
1 Back Up the Virtual Appliance Data
2 Prepare to Upgrade the Virtual Appliance
3 Upgrade with the OVA File
4 Verify the Deployment Status
5 Restore Data in the Virtual Appliance
6 Generate the Hybrid Configuration File
7 Configure the Standalone Controller for Hybrid Connectivity
8 Restart the Standalone Controller

Back Up the Virtual Appliance Data

Back up Virtual Appliance data before upgrading virtual machines in VMware ESXi.

Back up Virtual Appliance data before upgrading virtual machines in VMware ESXi.

  1. Download the Virtual Appliance OVA file:
    1. Log in to the Downloads portal using your credentials.
    2. In Downloads, select the Virtual Appliance tab.
    3. Select OVA from the Type list.
    4. Select Download.
  2. Shut down the Secure Application, Anomaly Detection, and remaining services. Then, the operators.
    CODE 
    appdcli stop secureapp
appdcli stop aiops
appdcli stop [services]
appdcli stop appd
<<Wait for pods to terminate before deleting operators>>
appdcli stop operators
    Note: Ensure that no pods are in the Running state before starting the data backup.
  3. Generate a backup file.
  4. Optional: Create a migration script for custom actions. See Migrate Custom Actions.
    Note: This step applies only if you are upgrading Virtual Appliance earlier than 26.1.0 version

Generate a Backup File

Generate a backup file and save it outside the Virtual Appliance cluster.

Generate a backup file and save it outside the Virtual Appliance cluster.
  1. Run the backup command:
    CODE 
    appdcli platform backup
  2. Copy the generated <backup_tar> file outside the cluster. If you have custom certificates, copy them to the same external location.

Prepare to Upgrade the Virtual Appliance

Prepare the Virtual Appliance virtual machines before starting the upgrade process.

Prepare the Virtual Appliance virtual machines before starting the upgrade process.

  1. Power off the virtual machines running the Kubernetes cluster.
  2. Rename the existing virtual machines so original names can be assigned to the new virtual machines
  3. For each existing virtual machine, select Edit Settings. In Virtual Hardware, expand Hard Disk 2 and record the Disk File name.
  4. Detach the data disk from each existing virtual machine. In Virtual Hardware > Hard Disk 2 > , select Remove Device.
CAUTION: Do not delete the preserved data-disk files. The disks detached in this procedure are attached to the new virtual machines during the upgrade.

Upgrade with the OVA File

Upgrade the Virtual Appliance by deploying new virtual machines from the OVA file.

  1. Deploy three virtual machines by using the upgrade OVA file.
  2. Remove Hard Disk 2 from each new virtual machine.
  3. Add an Existing Hard Disk to each new virtual machine and select the corresponding disk file recorded from the previous virtual machine.
  4. Enable VMware tools for OVF properties: select Configure > vApp Options > Edit > OVF Tools > VMware Tools
  5. Power on the new virtual machines.

Verify the Deployment Status

Verify the deployment status of Virtual Appliance virtual machines and services in VMware Esxi.

Verify the deployment of virtual machines:

  1. Verify whether the deployment of virtual machines is successful:
    CODE 
    appdctl show boot
    Note: Ensure that the services in each node is active. If any service appears as Failed, restart that virtual machine. You might have to redeploy the virtual machine if it is still failing.
  2. Create a three-node cluster:
    CODE 
    appdctl cluster init <Node-2-IP> <Node-3-IP>
  3. Verify node and MicroK8s status:
    CODE 
    appdctl show cluster
microk8s status
    Confirm that RUNNING is true for all three voter nodes.
    Note: You must re-login to the terminal if the following error appears:
    CODE 
    Insufficient Permissions to Access Microk8s

Restore Data in the Virtual Appliance

Restore backed-up data to the Virtual Appliance after completing a VMware ESXi upgrade.

  1. Copy the previously generated <backup_tar> file to one cluster node. See Prerequisites.
  2. Restore the persistent-volume specifications.
    CODE 
    appdcli platform restore <backup_tar>
  3. Respond to the prompt for self-signed ingress CA certificates:
    • Choose 0 if certificates are configured for DNS.
    • Choose 1 to restore and configure the self-signed certificates.
  4. Verify that all PVCs are in the Bound state:
    CODE 
    kubectl get pvc -A
  5. Start the following services and operators:
    CODE 
    appdcli start appd [Profile]
appdcli start aiops [Profile]
appdcli start secureapp [Profile]
appdcli start [Services] [Profile]
    Note: Sometimes, the MySQL Router pods might fail to start. See MySQL Router Pods Fail to Start.
  6. Sync to running controller.
    CODE 
    appdcli controller actions-sync
After the upgrade, the cluster uses the existing data disk and becomes functional.

Generate the Hybrid Configuration File

Generate the hybrid configuration file when custom certificates are not used for Ingress and Kafka clusters.

Hybrid deployments only: Generate this file when custom certificates are not used for the Ingress and Kafka clusters

Ensure that you have the latest CA certificates obtained after installing services. If not, update the CA certificates and regenerate the hybrid configuration file after restarting the service.

  1. On the Virtual Appliance, generate the hybrid configuration:
    CODE 
    cd ~/appd-charts/utils
bash prepare-hybrid.sh

    Sample output:

    CODE 
    Writing Kafka CA Cert
Writing Schema registry CA Cert
Writing AnomalyDetectionKafkaSSLProducer
Writing AnomalyDetectionKafkaConsumerSSL
Writing configure script
Creating archive of hybrid config
hybrid-config/
hybrid-config/kafka-ca.crt
hybrid-config/AnomalyDetectionKafkaConsumerSSL
hybrid-config/configure.sh
hybrid-config/schema-registry-ca.crt
hybrid-config/AnomalyDetectionKafkaSSLProducer
Completed

    The script creates hybrid-config.tar, containing Kafka and Schema Registry CA certificates, producer and consumer SSL configuration, and configure.sh.

  2. Copy the hybrid-config.tar file to the standalone Controller.
  3. Extract and run the configuration script:
    CODE 
    tar -xvf hybrid-config.tar
  4. Change to the extracted configuration directory and run the script:
    CODE 
    cd ~/hybrid-config
bash configure.sh
  5. Enter the actual Controller home when prompted:
    CODE 
    Controller home [/opt/appdynamics/platform/product/controller]: /path/to/platform/controller
    Note:

    Virtual Appliance certificates expire after five years. Regenerate the hybrid configuration and copy updated certificates to the Classic Controller as part of certificate rotation.

    For more information on how to audit and rotate certificates in Virtual Appliance, see Manage Virtual Appliance Certificates.

  6. Restart the classic Controller.

Configure the Standalone Controller for Hybrid Connectivity

Complete these checks before running the script:

  • Extract the generated hybrid-config archive on the Controller host.
  • Run the script from inside the extracted hybrid-config directory.

  • Identify the actual Controller home: The directory containing db/, tools/, and appserver/.

    Tip: /home/appdynamics/appdynamics/platform/controller. The suggested default is /opt/appdynamics/platform/product/controller, but the installation path can vary.
  • Obtain the DNS name configured as hybrid.mysql.dbHost in the Virtual Appliance.
  • Use an account with permission to update the Controller and MySQL certificate files.

Controller home example: /home/appdynamics/appdynamics/platform/controller. The suggested default is /opt/appdynamics/platform/product/controller, but the installation path can vary.

  1. Change to the extracted configuration directory and run the script:
    CODE 
    cd ~/hybrid-config
bash configure.sh
  2. Enter the actual Controller home when prompted:
    CODE 
    Controller home [/opt/appdynamics/platform/product/controller]: /path/to/platform/controller
    Note: If the Controller home is rejected, verify that the directory contains db/db.cnf.
  3. Review the proposed changes, then enter y to continue:
    CODE 
    This operation will:
  - Replace the Kafka and Schema Registry client truststores
  - Update the Controller CA truststore
  - Write obfuscated Kafka producer and consumer configuration
  - Replace the MySQL server certificate after creating a backup

Continue? [y/N]: y
  4. Confirm the reachable Controller address.
    The script detects a local IPv4 address. Confirm that the Virtual Appliance cluster can reach this address. Enter n to provide a different address.
    PYTHON 
    Detected Controller IP: 192.0.2.10
Is this address reachable from the Virtual Appliance cluster? [Y/n]: y
  5. Provide the hybrid database DNS name.
    Enter the exact hostname configured as hybrid.dbHost. The script adds this value to the MySQL certificate Subject Alternative Name (SAN).
    CODE 
    DNS name [192.0.2.10.nip.io]: controller.example.com

Keystore password behavior: The script first tries the standard keystore password, changeit. If that password cannot open the existing Controller CA truststore, the script prompts securely for the configured password. Typed password characters are not displayed.

The selected password is used consistently for:

  • The Kafka client truststore.
  • The Schema Registry client truststore.
  • The existing Controller CA truststore.
  • The truststore-password fields in the obfuscated producer and consumer configurations.
Note: If the keystore password is rejected, provide the password currently configured for appserver/jetty/etc/cacerts.jks.

Changes made by the script:

Item Location relative to Controller home Action
Kafka client truststore pi-kafka-ssl-config/kafka.client.truststore.jks Created or replaced using kafka-ca.crt.
Schema Registry client truststore pi-kafka-ssl-config/schema-registry.client.truststore.jks Created or replaced using schema-registry-ca.crt.
Controller CA truststore appserver/jetty/etc/cacerts.jks Replaces the k8s-cluster alias with schema-registry-ca.crt.
Kafka client configurations pi-kafka-ssl-config/AnomalyDetectionKafka* Writes obfuscated producer and consumer configuration.
MySQL server certificate Database data directory from db/db.cnf Generates a 365-day certificate containing the supplied DNS SAN.
Before replacement, the script creates timestamped backups of the Controller CA truststore and the existing MySQL server certificate. The final summary displays the complete backup paths.
Note: Use the timestamped backup paths printed by the script when following an approved rollback procedure.

If the configuration is successful, the following message will be displayed:

CODE 
Hybrid Controller configuration completed successfully.
Important:

After the script completes successfully, restart the Controller and its MySQL service. The Controller truststore and MySQL certificate changes do not take effect until the affected services are restarted.

Restart the Standalone Controller

Restart the Standalone Controller and MySQL service after completing configuration changes.

  • Run the following command to restart the standalone Controller:
    CODE 
    cd ~/appdynamics/platform/product/controller/bin/
./controller.sh stop
./controller.sh start

    This command also restarts the MySQL service.

    Alternatively, you can restart the controller from Enterprise Console. See Start or Stop the Controller.

  • For high-availability scenario, you must use Enterprise Console to restart the controller. See Start or Stop the Controller.