Prepare your Environment

Before installing a Hybrid Deployment, prepare your environment by ensuring secure and seamless communication between the Virtual Appliance and your On-Premises Controller. Collect and copy all required artifacts including certificates and keys for the Ingress Controller, Splunk AppDynamics オンプレミス Controller, and Database as described in the following sections.

This enables the Virtual Appliance to securely connect, authenticate, and establish encrypted communication channels with the Splunk AppDynamics オンプレミス Controller for reliable operation.

Create a Three-Node Cluster

Verify the boot status of each node of the cluster:

appdctl show boot

Note:

Ensure the status of the services in each node appears as Success. Else, restart the virtual machine that failed. If it is still failing, you might have to redeploy the virtual machine.
Ensure to configure the same time on all the cluster nodes.

Sample output:

 NAME             | STATUS    | ERROR
------------------+-----------+-------
 enable-time-sync | Succeeded | --
 firewall-setup   | Succeeded | --
 hostname         | Succeeded | --
 microk8s-setup   | Succeeded | --
 ssh-setup        | Succeeded | --
 storage-setup    | Succeeded | --
 cert-setup       | Succeeded | --

Run the following command in the primary node and specify the IP address of the peer nodes:
```
cd /home/appduser
appdctl cluster init <Node-2-IP> <Node-3-IP>
```
Run the following command to verify the node status:
```
appdctl show cluster
microk8s status
```
Ensure that the output displays the Running status as true for the nodes that are part of the cluster.
Sample Output
```
 NODE           | ROLE  | RUNNING 
----------------+-------+---------
 10.0.0.1:19001 | voter | true    
 10.0.0.2:19001 | voter | true    
 10.0.0.3:19001 | voter | true
```
Note:
- Ensure that there are at least two nodes are active in your cluster. If any two nodes become unavailable, there is a risk of data loss.
- You must re-login to the terminal if the following error appears;
  Insufficient Permissions to Access Microk8

Configure DNS Entries

Verify DNS resolution for the Splunk AppDynamics components. The following script helps to identify common network configuration issues that might cause communication issues in Virtual Appliance. It checks if specific host names can be resolved to IP addresses, which is crucial for the proper functioning of the Controller and its services.

Ensure that the globals.yaml.gotmpl contains the primary node IP address of Virtual Appliance. Edit the globals.yaml.gotmpl file as follows:

Log into the console of the primary node.
Navigate to the following folder:
```
cd /var/appd/config
```
Edit the globals.yaml.gotmpl file and update dnsNames and dnsDomain.
```
vi globals.yaml.gotmpl
```
List the Ingress domain names that you require to configure for the Virtual Appliance.
Note:
- Include local host for appdcli to access the cluster. If the domain names are unavailable, specify <nodeip>.nip.io for each cluster. You must also include the dnsDomain as a value, in addition to other specified values.
- Some network policies might block the IP address that contains x.x.x.x.nip.io. In such scenarios, update the /etc/hosts file. See, Update DNS Configuration for an Air-Gapped Environment.
- If you are using domain names for your Virtual Appliance, comment or delete the range split function. And, comment the localhost.
```
dnsNames: &dnsNames
# - localhost
  - 10.0.0.1.nip.io
  - 10.0.0.2.nip.io
  - 10.0.0.3.nip.io
  - appd.example.com
# If you are using Virtual IP address for your standalone Controller, comment or delete the range split function.
{{ range split " " $internalIPs }} {{ printf " - %s.%s" . "nip.io" }}
{{ end }}
```

Save the following script on the console of your primary Virtual Appliance node as dnsinfo.sh in /var/appd/config.

Note: If you are running this script for the first time, copy the code for plain YAML. If you are running this script after installing the services, copy the code for encrypted YAML.

Plain YAML

#!/bin/bash
set -euo pipefail

# Assuming /var/appd/config/secrets.yaml is now a plain, unencrypted YAML file.
# The '.encrypted' extension and 'helm secrets decrypt' command are removed.
TENANT=$(yq .hybrid.controller.tenantAccountName /var/appd/config/secrets.yaml)
CONTROLLER_DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.hybrid.controller.domainName')
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Controller domain name should be \'${CONTROLLER_DNS_DOMAIN}\'
 
echo
for server_name in "$CONTROLLER_DNS_DOMAIN" "${DNS_DOMAIN}" "${TENANT}.${CONTROLLER_DNS_DOMAIN}" "${TENANT}-tnt-con.${DNS_DOMAIN}" "${TENANT}-tnt-con.${CONTROLLER_DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please add DNS entry for ${server_name} for controller host IP, VA is not able to resolve it currently"
  fi
done
for server_name in "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please double-check on standalone controller that DNS can resolve entry for ${server_name} as VA ingress IP"
  fi
done

Encrypted YAML

#!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
CONTROLLER_DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.hybrid.controller.domainName')
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Controller domain name should be \'${CONTROLLER_DNS_DOMAIN}\'
 
echo
for server_name in "$CONTROLLER_DNS_DOMAIN" "${DNS_DOMAIN}" "${TENANT}.${CONTROLLER_DNS_DOMAIN}" "${TENANT}-tnt-con.${DNS_DOMAIN}" "${TENANT}-tnt-con.${CONTROLLER_DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please add DNS entry for ${server_name} for controller host IP, VA is not able to resolve it currently"
  fi
done
for server_name in "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please double-check on standalone controller that DNS can resolve entry for ${server_name} as VA ingress IP"
  fi
done

Run dnsinfo.sh.

bash ./dnsinfo.sh

Sample Output:

Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Controller domain name should be 'controller.mycompany.com'

Please add DNS entry for controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for va.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1.controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1-tnt-con.va.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1-tnt-con.controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please double-check on standalone controller that DNS can resolve entry for customer1.auth.va.mycompany.com as VA ingress IP
Please double-check on standalone controller that DNS can resolve entry for customer1-tnt-authn.va.mycompany.com as VA ingress IP

Enable Self-Monitoring Capabilities

You can enable self-monitoring capabilities for Virtual Appliance. To achieve this, edit the enableClusterAgent parameter to true in the global.yaml.gotmpl file.

By default, this parameter is set to true. This setting helps you monitor the CPU, Memory, and Network usage of your cluster nodes. See View Container Details

To disable self-monitoring capabilities, set this parameter to false.

enableClusterAgent: false

Note: After you enable or disable self-monitoring capabilities, you must restart the Virtual Appliance services to take effect.

(Optional) Configure Ingress Certificates

By default, the Ingress controller is installed with a fully-configured self-signed certificate. You may skip this step if the self-signed certificate provided by the Ingress controller meets your requirements.

If you require a CA signed certificate for the Ingress Controller, configure an SSL/TLS certificate for Splunk AppDynamics 自己ホスト型仮想アプライアンス by providing all the required host names. This ensures that all components and user access points of the Virtual Appliance are securely accessible. The following script generates a list of Subject Alternative Names (SANs) that would be required for a custom ingress certificate in Virtual Appliance. This is crucial for securing communication with the Virtual Appliance using HTTPS.

Ensure that you have the following files:

private key: private.key
signed public key: cert.crt
CA root chain: ca.crt

Note:

Ensure the ingess.key is in PEM plain text format.
The SAN of the server certificate in ingress.crt must include all the hostnames that are defined in the dnsNames section of the global.yaml.gotmpl file.
For Secure Application, ensure to include *.<DOMAIN-NAME> in the list and the certificates to include SAN aliases

Configure a custom ingress certificate if you want to include the CA signed certificate.

Copy the Ingress key to the Virtual Appliance.

scp <ingress.pem> appduser@<node-IP-address>:/var/appd/config/ingress.key

Copy the Ingress certificate to the Virtual Appliance.

scp <ingress.crt> appduser@<node-IP-address>:/var/appd/config/ingress.crt

Update the Standalone Controller TLS Certificate

Create a new standalone Controller certificate with all the required SANs. For more information, see Controller SSL and Certificates.
Note: If you use a load-balancer with a virtual IP address, ensure to copy the SSL certificates from the load-balancer instead of the standalone Controller to Virtual Appliance at: /var/appd/config.
For self-signed certificate, import the newly created certificate that has the DNS entries you added:
```
keytool -import -trustcacerts -alias s1as \
-file <certificate.crt> \
-keystore <ControllerInstallation>/platform/product/controller/appserver/jetty/etc/keystore.jks
```

Log in to your primary node of Virtual Appliance and save the following script on the node console as certs.sh in /var/appd/config:

Note: If you are running this script for the first time, copy the code for plain YAML. If you are running this script after installing the services, copy the code for encrypted YAML.

Plain YAML

#!/bin/bash
set -euo pipefail
TENANT=$(yq .hybrid.controller.tenantAccountName /var/appd/config/secrets.yaml)
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
DNS_NAMES=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsNames|join(" ")')

echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Virtual Appliance node names are: ${DNS_NAMES}

echo If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
for server_name in "$DNS_DOMAIN" "${TENANT}.${DNS_DOMAIN}" "*.${DNS_DOMAIN}" "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}" $DNS_NAMES; do
        echo "  ${server_name}"
done

Encrypted YAML

#!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
DNS_NAMES=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsNames|join(" ")')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Virtual Appliance node names are: ${DNS_NAMES}

echo If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
for server_name in "$DNS_DOMAIN" "${TENANT}.${DNS_DOMAIN}" "*.${DNS_DOMAIN}" "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}" $DNS_NAMES; do
        echo "  ${server_name}"
done

Run certs.sh to display the list of missing SANs.

bash ./certs.sh

Sample output displays the SANs that are missing in the custom ingress certificate:

Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Virtual Appliance node names are: localhost vanodename-1
If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
  va.mycompany.com
  customer1.va.mycompany.com
  *.va.mycompany.com
  customer1.auth.va.mycompany.com
  customer1-tnt-authn.va.mycompany.com
  localhost
  vanodename-1

Note: You can ignore the localhost and wildcard entries in the list.

Repeat the steps until all the required SANs are added to the Controller and imported them to the Virtual Appliance.
Note: Changes do not take affect until you restart the Controller.

Collect Certificates and Keys for Virtual Appliance

Virtual Appliance requires the certificates and keys from your Splunk AppDynamics オンプレミス Classic and its database to establish secure communication. If you use CA-signed certificates for Kafka services, you must copy them to the Virtual Appliance to maintain trusted connections and seamless operation within your environment.

Copy the SSL certificates from the standalone Controller to Virtual Appliance.

Controller: Go to the location where you have exported the certificate files. For example, <AppD Home>/controller/appserver/jetty/etc/
```
scp hybrid-controller-ca.crt appduser@<node-IP-address>:/var/appd/config/hybrid-controller-ca.crt
```
Note: Use the hybrid-controller-ca.crt certificate for HTTPS connections.
MySQL Database: Go to the database location. For example, <AppD Home>/controller/db/data.
```
scp ca.pem appduser@<node-IP-address>:/var/appd/config/hybrid-mysql-ca.crt
```
(Optional) Kafka: When you generate the CA certificates for Kafka, ensure to specify the Kafka IP addresses. These IP addresses are the same node IP addresses.
Copy a Kafka key to the Virtual Appliance.
```
scp <kafka.key> appduser@<node-IP-address>:/var/appd/config/hybrid-kafka.key
```
Copy a Kafka certificate to the Virtual Appliance.
```
scp <kafka.crt> appduser@<node-IP-address>:/var/appd/config/hybrid-kafka.crt
```

Complete the following steps to prepare the environment:

Log into the console of the primary node using the appduser credentials.This node is considered as the primary node after you create a three-node cluster.
Navigate to the following folder:
```
cd /var/appd/config
```
Edit the globals.yaml.gotmpl file with the required configuration. For more information, see globals.yaml.gotmpl
```
vi globals.yaml.gotmpl
```
Ensure the Virtual Appliance has network access, secure communications with SSL certificates, and configured database connections.
1. Enter the Controller key in license.controllerKey .
  Note: To find the Controller key,
  1. Log in to your Classic Splunk AppDynamics オンプレミス Controller UI.
  2. Navigate to Licenses > Account.
  3. In Access Key, click Show and copy the Controller key.
  Specify the license file that you require to apply on your Virtual Appliance.
  By default, the globals.yaml.gotmpl file has random UUIDs for controllerKey and eumKey. In case of Standard deployment, if you wish to generate new UUIDs for controllerkey and eumkey, run the gen-uuid.sh script from the helm charts folder (/home/appduser/appd-charts/utils).
  Specify the generated UUIDs in the license section for service and agent authorization.
  license: file: | {{ if isFile "/var/appd/config/license.lic" }} {{ readFile "/var/appd/config/license.lic" | indent 4 }} {{ end }} controllerKey: <Controller Key> eumKey: <EUM Key>
2. Verify the account name in the appdController.tenantAccountName sections. See appdController.
3. In the hybrid section, set the enable parameter to true.
  If you are using CA-signed certificates for Kafka services, update the hybrid.kafka section with required details.
  By default, this parameter is set to false.
  Enable this parameter to true to leverage your current Controller, Events Service, and End User Monitoring components from Splunk AppDynamics On-Premises while installing Anomaly Detection and Secure Application Services in your Kubernetes cluster.
  When set to true, the Controller and MySQL settings refer to an existing deployment of Controller.
  - Set the Controller domain name.
    Note:
    If you have set up the Virtual Appliance without a load-balancer or virtual IP address, the connection will not automatically switch to the active node during high-availability failover.
    Therefore, update the IP address, edit hybrid.controller.domainName and hybrid.mysql.dbhost in the global.yaml.gotmpl file after failover.
    If you have set up the Virtual Appliance with a load-balancer (virtual IP address), enter the load-balancer's domain name and port in the hybrid.controller.domainName, hybrid.controller.port, and hybrid.mysql.dbhost sections of the global.yaml.gotmpl file.
    This domain name should resolve to the load-balancer's virtual IP address.
  - Configure the port to access the standalone Controller.
  - If the TLS is enabled in the Controller
    Set the sslEnabled field to true .
    Upload the Controller CA certificates for the standalone Controller in /var/appd/config/hybrid-controller-ca.crt if it is absent.
  - Configure the MySQL host. It is the same host as the Controller domain.
  - Configure a port for the standalone Controller database.
  - Enter the MySQL CA certificates that you copied from the On-Premises Classic deployment.
    Note: Ensure that the filename is correct in hybrid.mysql.mysqlCaCertsFile parameter.
  - (Optional) Enter CA certificates for Kafka services.
    Note:
    Ensure that the filename is correct in hybrid.kafka.certFile parameter.
    When you generate the CA certificates for Kafka, ensure to specify the Kafka IP addresses. These IP addresses are the same node IP addresses.
    Example IP Addresses:
    ipAddresses: - 10.0.0.1 - 10.0.0.2 - 10.0.0.3
```
hybrid:
	enable: false
	controller:
		domainName: controller.nip.io
		port: 8181
		sslEnabled: true
{{ if isFile "/var/appd/config/hybrid-controller-ca.crt" }}
	controllerCaCertsFile: {{ readFile "/var/appd/config/hybrid-controller-ca.crt" | b64enc | quote }}
{{ end }}
mysql:
	dbHost: controller.nip.io
	dbPort: 3388
{{ if isFile "/var/appd/config/hybrid-mysql-ca.crt" }}
	mysqlCaCertsFile: {{ readFile "/var/appd/config/hybrid-mysql-ca.crt" | b64enc | quote }}
{{ end }}
kafka:
defaultCert: true
{{ if isFile "/var/appd/config/hybrid-kafka.key" }}
	keyFile: {{ readFile "/var/appd/config/hybrid-kafka.key" | b64enc | quote }}
{{ end }}
{{ if isFile "/var/appd/config/hybrid-kafka.crt" }}
	certFile: {{ readFile "/var/appd/config/hybrid-kafka.crt" | b64enc | quote }}
{{ end }}
schemaregistry:
	externalUrl: https://<domain_name>/schemaregistry
```
Edit the secrets.yaml file to configure secrets in the Virtual Appliance. For more information, see secrets.yaml
1. Update the Controller and MySQL credentials.
  Provide the Controller and MySQL credentials.
  hybrid: controller: controllerKey: <Controller-Key-Value> tenantAccountName: customer1 rootUsername: root rootPassword: welcome rootAccountname: system adminUsername: admin adminPassword: welcome mysql: dbUser: secapp dbPassword: changeit
2. (Optional) Edit the usernames and passwords of the Splunk AppDynamics services.
Update the firewall rules to allow the standalone Controller to access the Kafka ports in the Kubernetes cluster. Update the firewall rule in each cluster node.
```
sudo ufw allow AppdNodePorts
```