Prepare Your Environment

Prepare certificates, keys, and environment requirements before installing a hybrid deployment.

Before installing a Hybrid Deployment, prepare your environment by ensuring secure and seamless communication between the Virtual Appliance and your On-Premises Controller. Collect and copy all required artifacts including certificates and keys for the Ingress Controller, Splunk AppDynamics On-Premises Controller, and Database as described in the following sections.

This enables the Virtual Appliance to securely connect, authenticate, and establish encrypted communication channels with the Splunk AppDynamics On-Premises Controller for reliable operation.

Overview of hybrid deployment

Create a Three-Node Cluster

Create a three-node Virtual Appliance cluster from the primary node.

  1. Log in to the primary node console.
  2. Verify the boot status of each node of the cluster:
    CODE 
    appdctl show boot
    Note:

    • Ensure the status of the services in each node appears as Success. Else, restart the virtual machine that failed. If it is still failing, you might have to redeploy the virtual machine.

    • Ensure to configure the same time on all the cluster nodes.

    Sample output:

    CODE 
    NAME             | STATUS    | ERROR
------------------+-----------+-------
 enable-time-sync | Succeeded | --
 firewall-setup   | Succeeded | --
 hostname         | Succeeded | --
 microk8s-setup   | Succeeded | --
 ssh-setup        | Succeeded | --
 storage-setup    | Succeeded | --
 cert-setup       | Succeeded | --
  3. Run the following command in the primary node and specify the IP address of the peer nodes:
    CODE 
    cd /home/appduser
appdctl cluster init <Node-2-IP> <Node-3-IP>
  4. Run the following command to verify the node status:
    CODE 
    appdctl show cluster
microk8s status

    Ensure that the output displays the Running status as true for the nodes that are part of the cluster.

    Sample Output

    CODE 
    NODE           | ROLE  | RUNNING 
----------------+-------+---------
 10.0.0.1:19001 | voter | true    
 10.0.0.2:19001 | voter | true    
 10.0.0.3:19001 | voter | true
    Note:
    • Ensure that there are at least two nodes are active in your cluster. If any two nodes become unavailable, there is a risk of data loss.
    • You must re-login to the terminal if the following error appears;
      CODE 
      Insufficient Permissions to Access Microk8

Configure DNS Entries

Configure DNS entries required for hybrid deployment communication.

Verify DNS resolution for the Splunk AppDynamics components. The following script helps identify common network configuration issues that might cause communication issues in Virtual Appliance. It checks if specific host names can be resolved to IP addresses, which is crucial for the proper functioning of the Controller and its services.

Ensure that the globals.yaml.gotmpl contains the primary node IP address of Virtual Appliance. Edit the globals.yaml.gotmpl file as follows:

  1. Log in to the console of the primary node.

  2. Navigate to the following folder:
    CODE 
    cd /var/appd/config
  3. Edit the globals.yaml.gotmpl file and update dnsNames and dnsDomain.
    CODE 
    vi globals.yaml.gotmpl

    List the Ingress domain names that you require to configure for the Virtual Appliance.

    Note:
    • Include localhost for appdcli to access the cluster. If the domain names are unavailable, specify <nodeip>.nip.io for each cluster. You must also include the dnsDomain as a value, in addition to other specified values.
    • Some network policies might block the IP address that contains x.x.x.x.nip.io. In such scenarios, update the /etc/hosts file. See, Update DNS Configuration for an Air-Gapped Environment.
    • If you are using domain names for your Virtual Appliance, comment or delete the range split function. Then, comment the localhost.
    JSON 
    dnsNames: &dnsNames
# - localhost
  - 10.0.0.1.nip.io
  - 10.0.0.2.nip.io
  - 10.0.0.3.nip.io
  - appd.example.com
# If you are using Virtual IP address for your standalone Controller, comment or delete the range split function.
{{ range split " " $internalIPs }} {{ printf " - %s.%s" . "nip.io" }}
{{ end }}
  1. Save the following script on the console of your primary Virtual Appliance node as dnsinfo.sh in /var/appd/config.
    Note: If you are running this script for the first time, copy the code for plain YAML. If you are running this script after installing the services, copy the code for encrypted YAML.
    Plain YAML
    CODE 
    #!/bin/bash
set -euo pipefail

# Assuming /var/appd/config/secrets.yaml is now a plain, unencrypted YAML file.
# The '.encrypted' extension and 'helm secrets decrypt' command are removed.
TENANT=$(yq .hybrid.controller.tenantAccountName /var/appd/config/secrets.yaml)
CONTROLLER_DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.hybrid.controller.domainName')
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Controller domain name should be \'${CONTROLLER_DNS_DOMAIN}\'
 
echo
for server_name in "$CONTROLLER_DNS_DOMAIN" "${DNS_DOMAIN}" "${TENANT}.${CONTROLLER_DNS_DOMAIN}" "${TENANT}-tnt-con.${DNS_DOMAIN}" "${TENANT}-tnt-con.${CONTROLLER_DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please add DNS entry for ${server_name} for controller host IP, VA is not able to resolve it currently"
  fi
done
for server_name in "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please double-check on standalone controller that DNS can resolve entry for ${server_name} as VA ingress IP"
  fi
done
    Encrypted YAML
    CODE 
    #!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
CONTROLLER_DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.hybrid.controller.domainName')
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Controller domain name should be \'${CONTROLLER_DNS_DOMAIN}\'
 
echo
for server_name in "$CONTROLLER_DNS_DOMAIN" "${DNS_DOMAIN}" "${TENANT}.${CONTROLLER_DNS_DOMAIN}" "${TENANT}-tnt-con.${DNS_DOMAIN}" "${TENANT}-tnt-con.${CONTROLLER_DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please add DNS entry for ${server_name} for controller host IP, VA is not able to resolve it currently"
  fi
done
for server_name in "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please double-check on standalone controller that DNS can resolve entry for ${server_name} as VA ingress IP"
  fi
done
  2. Run dnsinfo.sh.
    CODE 
    bash ./dnsinfo.sh
    Sample Output:
    CODE 
    Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Controller domain name should be 'controller.mycompany.com'

Please add DNS entry for controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for va.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1.controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1-tnt-con.va.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1-tnt-con.controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please double-check on standalone controller that DNS can resolve entry for customer1.auth.va.mycompany.com as VA ingress IP
Please double-check on standalone controller that DNS can resolve entry for customer1-tnt-authn.va.mycompany.com as VA ingress IP

Enable Self-Monitoring Capabilities

Enable self-monitoring capabilities for the Virtual Appliance by updating the global configuration file.

You can enable self-monitoring capabilities for Virtual Appliance. To achieve this, edit the enableClusterAgent parameter to true in the global.yaml.gotmpl file.

By default, this parameter is set to true. This setting helps you monitor the CPU, Memory, and Network usage of your cluster nodes. See View Container Details

To disable self-monitoring capabilities, set this parameter to false.
CODE 
enableClusterAgent: false
Note: After you enable or disable self-monitoring capabilities, you must restart the Virtual Appliance services to take effect.

Configure Ingress Certificates (Only for SSL Certificates)

Configure custom ingress certificates and required SANs for the Virtual Appliance.

By default, the Ingress controller is installed with a fully configured self-signed certificate.
Note: You may skip this step if the self-signed certificate provided by the Ingress Controller meets your requirements.

If you require a CA-signed certificate for the Ingress Controller, configure an SSL/TLS certificate for Splunk AppDynamics Self Hosted Virtual Appliance by providing all the required hostnames. This ensures that all components and user access points of the Virtual Appliance are securely accessible. The following script generates a list of Subject Alternative Names (SANs) that would be required for a custom ingress certificate in Virtual Appliance. This is crucial for securing communication with the Virtual Appliance using HTTPS.

Ensure that you have the following files:

Private key

Filename must be ingress.pem. Ensure that ingress.pem is in PEM plain text format.

PEM certificate bundle

Filename must be ingress.crt. Ensure the ingress.pem is in PEM plain text format.

The ingress.crt PEM certificate bundle requires a particular order for the PEM-encoded certificates:

  • leaf/server certificate
  • intermediate CA certificate(s)
  • root CA certificate

The SAN of the server certificate in ingress.pem must include all the hostnames that are defined in the dnsNames section of the global.yaml.gotmpl file.

For Secure Application, ensure that you include *.<DOMAIN-NAME> and all required SAN aliases in the certificate's Subject Alternative Name (SAN) list.

Configure a custom ingress certificate if you want to include the CA-signed certificate.
  1. Copy the private key to the Virtual Appliance.
    CODE 
    scp ingress.pem appduser@<node-IP-address>:/var/appd/config/ingress.key
  2. Copy PEM certificate bundle to the Virtual Appliance.
    CODE 
    scp ingress.crt appduser@<node-IP-address>:/var/appd/config/ingress.crt

Update the Standalone Controller TLS Certificate

Update the standalone Controller TLS certificate and import required SANs into the Virtual Appliance.

  1. Create a new standalone Controller certificate with all the required SANs. For more information, see Controller SSL and Certificates.
    Note: If you use a load-balancer with a virtual IP address, ensure to copy the SSL certificates from the load-balancer instead of the standalone Controller to Virtual Appliance at: /var/appd/config.

    For self-signed certificate, import the newly created certificate that has the DNS entries you added:

    PYTHON 
    keytool -import -trustcacerts -alias s1as \
-file <certificate.crt> \
-keystore <ControllerInstallation>/platform/product/controller/appserver/jetty/etc/keystore.jks
  2. Log in to your primary node of Virtual Appliance and save the following script on the node console as certs.sh in /var/appd/config:
    Note: If you are running this script for the first time, copy the code for plain YAML. If you are running this script after installing the services, copy the code for encrypted YAML.
    Plain YAML
    JSON 
    #!/bin/bash
set -euo pipefail
TENANT=$(yq .hybrid.controller.tenantAccountName /var/appd/config/secrets.yaml)
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
DNS_NAMES=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsNames|join(" ")')

echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Virtual Appliance node names are: ${DNS_NAMES}

echo If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
for server_name in "$DNS_DOMAIN" "${TENANT}.${DNS_DOMAIN}" "*.${DNS_DOMAIN}" "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}" $DNS_NAMES; do
        echo "  ${server_name}"
done
    Encrypted YAML
    JSON 
    #!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
DNS_NAMES=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsNames|join(" ")')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Virtual Appliance node names are: ${DNS_NAMES}

echo If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
for server_name in "$DNS_DOMAIN" "${TENANT}.${DNS_DOMAIN}" "*.${DNS_DOMAIN}" "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}" $DNS_NAMES; do
        echo "  ${server_name}"
done
  3. Run certs.sh to display the list of missing SANs.
    CODE 
    bash ./certs.sh
    Sample output displays the SANs that are missing in the custom ingress certificate:
    CODE 
    Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Virtual Appliance node names are: localhost vanodename-1
If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
  va.mycompany.com
  customer1.va.mycompany.com
  *.va.mycompany.com
  customer1.auth.va.mycompany.com
  customer1-tnt-authn.va.mycompany.com
  localhost
  vanodename-1
    Note: You can ignore the localhost and wildcard entries in the list.
  4. Repeat the steps until all the required SANs are added to the Controller and imported into the Virtual Appliance.
    Note: Changes do not take effect until you restart the Controller.

Collect Certificates and Keys for Virtual Appliance

Collect certificates and keys required for secure communication between the Virtual Appliance and existing components.

Virtual Appliance requires the certificates and keys from your Splunk AppDynamics On-Premises the classic Controller and its database to establish secure communication. If you use CA-signed certificates for Kafka services, you must copy them to the Virtual Appliance to maintain trusted connections and seamless operation within your environment.
Copy the SSL certificates from the standalone Controller to Virtual Appliance.

  • Controller: Go to the location where you have exported the certificate files. For example, <AppD Home>/controller/appserver/jetty/etc/

    CODE 
    scp hybrid-controller-ca.crt appduser@<node-IP-address>:/var/appd/config/hybrid-controller-ca.crt
    Note: Use the hybrid-controller-ca.crt certificate for HTTPS connections.

  • MySQL Database: Go to the database location. For example, <AppD Home>/controller/db/data.

    CODE 
    scp ca.pem appduser@<node-IP-address>:/var/appd/config/hybrid-mysql-ca.crt

  • (Optional) Kafka: When you generate the CA certificates for Kafka, ensure to specify the Kafka IP addresses. These IP addresses are the same node IP addresses.

    Copy a Kafka key to the Virtual Appliance.

    CODE 
    scp <kafka.key> appduser@<node-IP-address>:/var/appd/config/hybrid-kafka.key

    Copy a Kafka certificate to the Virtual Appliance.

    CODE 
    scp <kafka.crt> appduser@<node-IP-address>:/var/appd/config/hybrid-kafka.crt

Complete the following steps to prepare the environment:

  1. Log in to the console of the primary node using the appduser credentials. This node is considered the primary node after you create a three-node cluster.
  2. Navigate to the following folder:
    CODE 
    cd /var/appd/config
  3. Edit the globals.yaml.gotmpl file with the required configuration. For more information, see globals.yaml.gotmpl
    CODE 
    vi globals.yaml.gotmpl

    Ensure the Virtual Appliance has network access, secure communications with SSL certificates, and configured database connections.

    1. Enter the Controller key in license.controllerKey .
      Note: To find the Controller key,
      1. Log in to your Classic Splunk AppDynamics On-Premises Controller UI.
      2. Navigate to Licenses > Account.
      3. In Access Key, click Show and copy the Controller key.

      Specify the license file that you require to apply on your Virtual Appliance.

      By default, the globals.yaml.gotmpl file has random UUIDs for controllerKey and eumKey. In case of Standard deployment, if you wish to generate new UUIDs for controllerKey and eumKey, run the gen-uuid.sh script from the helm charts folder (/home/appduser/appd-charts/utils).

      Specify the generated UUIDs in the license section for service and agent authorization.

      JSON 
      license:
	file: |
{{ if isFile "/var/appd/config/license.lic" }}
{{ readFile "/var/appd/config/license.lic" | indent 4 }}
{{ end }}
	controllerKey: <Controller Key>
	eumKey: <EUM Key>
    2. Verify the account name in the appdController.tenantAccountName sections. See appdController.
    3. In the hybrid section, set the enable parameter to true.
      If you are using CA-signed certificates for Kafka services, update the hybrid.kafka section with required details.

      By default, this parameter is set to false.

      Enable this parameter to true to leverage your current Controller, Events Service, and End User Monitoring components from Splunk AppDynamics On-Premises while installing Anomaly Detection and Secure Application Services in your Kubernetes cluster.

      When set to true, the Controller and MySQL settings refer to an existing deployment of Controller.

      • Set the Controller domain name.
        Note:

        • If you have set up the Virtual Appliance without a load-balancer or virtual IP address, the connection will not automatically switch to the active node during high-availability failover.

          Therefore, update the IP address, edit hybrid.controller.domainName and hybrid.mysql.dbhost in the global.yaml.gotmpl file after failover.

        • If you have set up the Virtual Appliance with a load-balancer (virtual IP address), enter the load-balancer's domain name and port in the hybrid.controller.domainName, hybrid.controller.port, and hybrid.mysql.dbhost sections of the global.yaml.gotmpl file.

          This domain name should resolve to the load-balancer's virtual IP address.

      • Configure the port to access the standalone Controller.
      • If the TLS is enabled in the Controller
        • Set the sslEnabled field to true .
        • Upload the Controller CA certificates for the standalone Controller in /var/appd/config/hybrid-controller-ca.crt if it is absent.
      • Configure the MySQL host. It is the same host as the Controller domain.
      • Configure a port for the standalone Controller database.
      • Enter the MySQL CA certificates that you copied from the On-Premises Classic deployment.
        Note: Ensure that the filename is correct in hybrid.mysql.mysqlCaCertsFile parameter.
      • (Optional) Enter CA certificates for Kafka services.
        Note:

        Ensure that the filename is correct in hybrid.kafka.certFile parameter.

        When you generate the CA certificates for Kafka, ensure to specify the Kafka IP addresses. These IP addresses are the same node IP addresses.

        Example IP Addresses:

        CODE 
        ipAddresses:
- 10.0.0.1
- 10.0.0.2
- 10.0.0.3
      JSON 
      hybrid:
	enable: false
	controller:
		domainName: controller.nip.io
		port: 8181
		sslEnabled: true
{{ if isFile "/var/appd/config/hybrid-controller-ca.crt" }}
	controllerCaCertsFile: {{ readFile "/var/appd/config/hybrid-controller-ca.crt" | b64enc | quote }}
{{ end }}
mysql:
	dbHost: controller.nip.io
	dbPort: 3388
{{ if isFile "/var/appd/config/hybrid-mysql-ca.crt" }}
	mysqlCaCertsFile: {{ readFile "/var/appd/config/hybrid-mysql-ca.crt" | b64enc | quote }}
{{ end }}
kafka:
defaultCert: true
{{ if isFile "/var/appd/config/hybrid-kafka.key" }}
	keyFile: {{ readFile "/var/appd/config/hybrid-kafka.key" | b64enc | quote }}
{{ end }}
{{ if isFile "/var/appd/config/hybrid-kafka.crt" }}
	certFile: {{ readFile "/var/appd/config/hybrid-kafka.crt" | b64enc | quote }}
{{ end }}
schemaregistry:
	externalUrl: https://<domain_name>/schemaregistry
  4. Edit the secrets.yaml file to configure secrets in the Virtual Appliance. For more information, see secrets.yaml
    1. Update the Controller and MySQL credentials.
      YAML 
      hybrid:
	controller:
		controllerKey: <Controller-Key-Value>
		tenantAccountName: customer1
		rootUsername: root
		rootPassword: welcome
		rootAccountname: system
		adminUsername: admin
		adminPassword: welcome
	mysql:
		dbUser: secapp
		dbPassword: changeit
    2. (Optional) Edit the usernames and passwords of the Splunk AppDynamics services.
  5. Update the firewall rules to allow the standalone Controller to access the Kafka ports in the Kubernetes cluster. Update the firewall rule on each cluster node.
    CODE 
    sudo ufw allow AppdNodePorts