Upgrade the Virtual Appliance in VMware

Upgrade the Virtual Appliance in VMware ESXi by deploying new virtual machines from an OVA file.

Upgrading the Virtual Appliance involves:

  • Backing up and deleting the hard disks of the existing virtual machines.
  • Deploying new virtual machines by using OVA image.
  • Restoring the backup to their hard disks.
  • Attaching the new hard disks to the older virtual machines

Splunk AppDynamics On-Premises Virtual Appliance provides the reference script that helps in upgrading the Virtual Appliance.

Standard Deployment
Sl No Steps
1 Back Up the Virtual Appliance Data
2 Prepare to upgrade the Virtual Appliance
3 Upgrade the Virtual Appliance with the OVA File
4 Verify the Deployment Status
5 Restore Data in the Virtual Appliance
6 Migrate EUM and Synthetic Services Data to MinIO
Hybrid Deployment
Sl No. Steps
1 Back Up the Virtual Appliance Data
2 Prepare to upgrade the Virtual Appliance
3 Upgrade the Virtual Appliance with the OVA File
4 Verify the Deployment Status
5 Restore Data in the Virtual Appliance
6 Migrate EUM and Synthetic Services Data to MinIO
7 Generate the Hybrid Configuration File
Note: This document contains links to VMware documentation. Splunk AppDynamics makes no representation as to the accuracy of VMware documentation because VMware controls its own documentation.

Back Up the Virtual Appliance Data

Back up Virtual Appliance data before upgrading virtual machines in VMware ESXi.

Follow these to back up the Virtual Appliance data:

  1. Download the Virtual Appliance OVA file:
    1. Log in to the Downloads portal using your credentials.
    2. In Downloads, select the Virtual Appliance tab.
    3. Select OVA from the Type list.
    4. Select Download.
    Note: The file available in the downloads portal is used for both installation and upgrade procedures.
  2. Shutdown the Secure Application, Anomaly Detection, and remaining services. Then, shutdown Splunk AppDynamics.
    CODE 
    appdcli stop secapp
appdcli stop aiops
appdcli stop [services]
appdcli stop appd
<<Wait for pods to terminate before deleting operators>>
appdcli stop operators
    Note:

    • Ensure that there are no pods in the running state before you start the data backup.

    • Sometimes, IOException error occurs in the Controller UI. See Troubleshoot Virtual Appliance Issues
    .
  3. Optional: Create a migration script for custom actions. See Migrate Custom Actions.
  4. Generate a backup file.

Migrate Custom Actions

Migrate custom actions to the updated path during a Virtual Appliance upgrade.

The custom actions path has changed from the previous release. Create and run the following script to migrate custom actions during the upgrade.

  1. Create a migration script for custom actions (custom-actions-migrate.sh) in /var/appd/config .
  2. Edit custom-actions.migrate.sh.
    CODE 
    vi /var/appd/config/custom-actions-migrate.sh
    JSON 
    #!/usr/bin/env bash

set -euo pipefail

# Configuration
OLD=/var/appd/data/custom-actions
NEW=/var/appd/config/custom-actions
ARCHIVE=${2:-custom-actions.tgz}

case "${1:-}" in
    backup)
        test -d "$OLD/actions" || { echo "No custom actions found at $OLD/actions"; exit 1; }
        
        sudo tar -C "$OLD" -czf "$ARCHIVE" actions
        echo "Backup created: $ARCHIVE"
        ;;

    restore)
        test -f "$ARCHIVE" || { echo "Backup file not found: $ARCHIVE"; exit 1; }
        
        sudo mkdir -p "$NEW"
        sudo tar -C "$NEW" -xzf "$ARCHIVE"
        
        # Set permissions
        sudo chown -R appduser:appduser "$NEW/actions"
        sudo find "$NEW/actions" -type d -exec chmod 755 {} \;
        sudo find "$NEW/actions" -type f -exec chmod 755 {} \;
        
        echo "Restored custom actions to $NEW/actions"
        ;;

    *)
        echo "Usage: $0 backup [custom-actions.tgz] | restore <custom-actions.tgz>"
        exit 1
        ;;
esac
  3. Change permissions of the migration script.
    CODE 
    chmod +x /var/appd/config/custom-actions-migrate.sh
  4. Sync custom actions to the running controller.
    CODE 
    appdcli controller actions_sync
  5. Create a custom action archive.
    CODE 
    ./custom-actions-migrate.sh backup custom-actions.tgz
Generate a Backup File

Generate a Backup File

Generate a backup file and save it outside the Virtual Appliance cluster.

Follow the steps to generate a backup file and save it outside the cluster:
  1. Run the backup command:
    CODE 
    appdcli run backup
    This command generates a <backup_tar> file with all the necessary data.
  2. Copy and paste the <backup_tar> file outside of the cluster. If you have custom certificates, copy them to the same location as the backup file.

Prepare to upgrade the Virtual Appliance

Prepare the Virtual Appliance virtual machines before starting the upgrade process.

Complete the following steps to prepare the Virtual Appliance upgrade:

  1. Power off the virtual machines that are running the Kubernetes cluster.
  2. Rename the existing virtual machines to use their names for the new virtual machines.
  3. Right click each virtual machine and select Edit Settings:
    1. In Virtual Hardware, expand Hard Disk 2.
    2. Note the Disk File name.
      Edit Settings
  4. Remove the data disk from the virtual machines.
    1. Right click on each virtual machine and select Edit Settings.
    2. In Virtual Hardware > Hard Disk2, select Remove Device from the more options.Remove Data Disk From the VM

Upgrade the Virtual Appliance with the OVA File

Upgrade the Virtual Appliance by deploying new virtual machines from the OVA file.

Complete the following steps to upgrade the Virtual Appliance by using an OVA file:

  1. Deploy three virtual machines by using the upgrade OVA file.
  2. Delete the data disk of the new virtual machine:
    1. Go to Edit Settings.
    2. Select Hard Disk 2.
    3. Remove device
  3. Attach the data disks of the previous virtual machines:
    1. Go to Edit Settings.
    2. Add a new device.
    3. Select Existing Hard Disk and specify the disk file name that you have noted earlier.
  4. Enable VMware tools in the virtual machines to configure OVF properties.
    1. Select th eConfigure tab.
    2. Select vApp Options and click Edit.
    3. In Edit vApp Options > OVF Tools, select VMware Tools. See Edit OVF properties for a virtual machine.
  5. Power on the new virtual machines.

Verify the Deployment Status

Verify the deployment status of Virtual Appliance virtual machines and services in VMware Esxi.

Verify the deployment of virtual machines:

  1. Verify whether the deployment of virtual machines is successful:
    CODE 
    appdctl show boot
    Note: Ensure that the services in each node is active. If any service appears as Failed, restart that virtual machine. You might have to redeploy the virtual machine if it is still failing.
  2. Create a three-node cluster:
    1. Run the following command in the primary node and specify the IP address of the peer nodes:
      CODE 
      appdctl cluster init <Node-2-IP> <Node-3-IP>
    2. Run the following command to verify the node status:
      CODE 
      appdctl show cluster
microk8s status
      Ensure that the output displays the Running status as true for the nodes that are part of the cluster.

      Sample Output

      CODE 
      NODE           | ROLE  | RUNNING
----------------+-------+---------
10.0.0.1:19001 | voter | true
10.0.0.2:19001 | voter | true
10.0.0.3:19001 | voter | true
      Note: You must re-login to the terminal if the following error appears:
      CODE 
      Insufficient Permissions to Access Microk8s

Restore Data in the Virtual Appliance

Restore backed-up data to the Virtual Appliance after completing a VMware ESXi upgrade.

  1. Copy the <backup_tar> file to one of the cluster nodes that is generated earlier. See Prerequisites.
  2. Run the following command to restore the persistent volumes specifications:
    CODE 
    appdcli run restore <backup_tar>
  3. Restore custom actions to a new location.
    CODE 
    ./custom-actions-migrate.sh restore custom-actions.tgz
  4. Restore self-signed CA ingress certificates.A prompt is displayed "Do you want to restore self-signed ingress certificates?" Enter 0 for No or 1 for Yes".
    • Choose 0 if you have configured the certificates for DNS.
    • Choose 1 to configure the certificates.
  5. Verify that the PVC are in the Bound state by running the following command:
    CODE 
    kubectl get pvc -A
  6. Start the following services and operators:
    1. Splunk AppDynamics Services.
      CODE 
      appdcli start appd [Profile]
    2. Anomaly Detection Service.
      CODE 
      appdcli start aiops [Profile]
    3. Secure Application Service.
      CODE 
      appdcli start secapp [Profile]
    4. Start the remaining services.
      CODE 
      appdcli start [Services] [Profile]
    Note: Sometimes, the MySQL Router pods might fail to start. See MySQL Router Pods Fail to Start.
  7. Sync to running controller.
    CODE 
    appdcli controller actions_sync
After the upgrade, the cluster uses the existing data disk and becomes functional.

Migrate EUM and Synthetic Services Data to MinIO

Migrate existing EUM and Synthetic services data to MinIO for the Virtual Appliance.

If you are running EUM and Synthetic services, you must migrate the existing data in these services to MinIO. To migrate data, you must run the migration script. Complete the following steps to enable data migration:
  1. Log in to the Virtual Appliance node and edit the globals.yaml.gotmpl file.
    CODE 
    cd /var/appd/config
vi globals.yaml.gotmpl
  2. Enable data migration for Minio, set the minio.migration.enable parameter to true.
    CODE 
    minio:
  migration:
    enable: true
  3. Sync the changes across all the cluster nodes.
    CODE 
    appdcli sync appd <profile>

Generate the Hybrid Configuration File

Generate the hybrid configuration file when custom certificates are not used for Ingress and Kafka clusters.

Generate the hybrid configuration file only if you are not using custom certificates for Ingress and Kafka clusters.

Ensure that you have the latest CA certificates obtained after installing services. If not, update the CA certificates and regenerate the hybrid configuration file after restarting the service.

  1. Run the script to connect the Kafka node ports with the standalone Controller:
    CODE 
    ~/appd-charts/utils$ bash prepare-hybrid.sh

    Sample output:

    CODE 
    Writing Kafka CA Cert
Writing Schema registry CA Cert
Writing AnomalyDetectionKafkaSSLProducer
Writing AnomalyDetectionKafkaConsumerSSL
Writing configure script
Creating archive of hybrid config
hybrid-config/
hybrid-config/kafka-ca.crt
hybrid-config/AnomalyDetectionKafkaConsumerSSL
hybrid-config/configure.sh
hybrid-config/schema-registry-ca.crt
hybrid-config/AnomalyDetectionKafkaSSLProducer
Completed

    This script generates the hybrid-config.tar file that includes the required configuration files.

  2. Copy the hybrid-config.tar file to your standalone Controller instance.
  3. Log in to the standalone Controller.
    1. Locate and extract the hybrid-config.tar file contents.
      CODE 
      tar -xvf hybrid-config.tar
    2. Run the configure.sh script to provision the Controller with certificates and secrets to connect to the Kubernetes cluster:

      The following script is extracted from the hybrid-config.tar file.

      CODE 
      $ ./configure.sh
      Output:
      CODE 
      Using controller home as /home/appdynamics/appdynamics and keystore password as changeit
Do you want to proceed? (y/n): y

Creating Kafka truststore
Certificate was added to keystore

Creating schema registry truststore
Certificate was added to keystore
Certificate was added to keystore

Writing obfuscated producer config
Writing obfuscated consumer config

Configure MySQL for SSL
Using controller ip as <IP>
Is this IP address correct? (y/n): y

DNS name should match the Virtual Appliance cluster hybrid dbHost configuration
Please enter the DNS name (default: 10.0.0.1.nip.io): <DNS_Name_of_Classic_On_Premises_Controller>

Using controller IP as 10.0.0.1
Updating certs at /home/appdynamics/appdynamics/platform/controller/db/data using CA key file ca-key.pem and cert file ca.pem
Certificate request self-signature ok
subject=CN = mysql
MySQL certificate generated successfully: server-cert.pem
Completed
    Note:

    Certificates in the Virtual Appliance expire after 60 days. To maintain connectivity, you must copy the updated certificates from the Virtual Appliance to your Splunk AppDynamics On-Premises Classic Controller. To do this, regenerate the hybrid configuration file on the Virtual Appliance and then copy it to your Standalone Controller.