Prepare the Virtual Appliance

Prepare certificates, keys, and environment requirements before installing a hybrid deployment.

Prepare certificates, keys, and environment requirements.

Before enabling hybrid mode, build the cluster and then establish name resolution and trust.

Create a Three-Node Cluster

Create a three-node Virtual Appliance cluster from the primary node.

  1. Log in to every node and verify the boot services.
  2. On the primary node, initialize the cluster using the two peer IP addresses.
  3. Verify that every member is ready, schedulable, and running.
    CODE 
    # Run on every node
appdctl show boot

    Every boot service reports Succeeded; all three nodes report READY=true, SCHEDULABLE=true, and RUNNING=true

    Sample Output:
    CODE 
    NAME              | STATUS    | ERROR 
-------------------+-----------+-------
 firewall-setup    | Succeeded | --    
 hostname          | Succeeded | --    
 netplan           | Succeeded | --    
 ssh-setup         | Succeeded | --    
 storage-setup     | Succeeded | --    
 cert-setup        | Succeeded | --    
 enable-time-sync  | Succeeded | --    
 microk8s-setup    | Succeeded | --    
 cloud-init-config | Succeeded | --
  4. Run this command on a primary node.
    CODE 
    # Run on the primary node
cd /home/appduser
appdctl cluster init <NODE_2_IP> <NODE_3_IP>
appdctl show cluster
microk8s status
    Warning: Keep at least two nodes active. If two nodes are unavailable, the cluster can lose data. If MicroK8s reports insufficient permissions, log out and back in before retrying.
    Sample Output:
    CODE 
    NODE             | INTERNAL IP   | K8S ROLE      | HA ROLE | READY | SCHEDULABLE
------------------+---------------+---------------+---------+-------+-------------
 ip-10-115-84-215 | 10.115.84.215 | control-plane | voter   | true  | true
 ip-10-115-85-239 | 10.115.85.239 | control-plane | voter   | true  | true
 ip-10-115-87-4   | 10.115.87.4   | control-plane | voter   | true  | true

Set Up the Hybrid Mode

Configure DNS
  1. Log in to you primary node and open the /var/appd/config/globals.yaml.gotmpl file.
  2. Replace all <domain_name> occurrences with Virtual Appliance domain name.

    Example dnsNames configuration

    CODE 
    dnsDomain: va.example.com
dnsNames: &dnsNames
  - localhost
  - va.example.com
  - va-node-1.example.com
  - va-node-2.example.com
  - va-node-3.example.com

    Verify whether the output displays the supplied domain names.

    Sample Output:
    CODE 
    Deployment mode: hybrid
Virtual Appliance tenant: 'e2e-customer'
Virtual Appliance domain: 'bf-manual-va.e2e.appd-test.com'
Controller tenant: 'e2e-customer'
Controller domain: 'bf-manual-cont-es-ec-controller.e2e.appd-test.com'
[OK] bf-manual-cont-es-ec-controller.e2e.appd-test.com
[OK] bf-manual-va.e2e.appd-test.com
[OK] e2e-customer.bf-manual-cont-es-ec-controller.e2e.appd-test.com
[OK] e2e-customer-tnt-con.bf-manual-va.e2e.appd-test.com
[OK] e2e-customer-tnt-con.bf-manual-cont-es-ec-controller.e2e.appd-test.com
[OK] localhost
[OK] bf-manual-va.e2e.appd-test.com
[OK] 10.115.84.215.nip.io
[OK] 10.115.85.239.nip.io
[OK] 10.115.87.4.nip.io
[OK] e2e-customer.auth.bf-manual-va.e2e.appd-test.com
[OK] e2e-customer-tnt-authn.bf-manual-va.e2e.appd-test.com

DNS verification completed: 0 unresolved host(s)
    Note: If registered names are not available, the source allows <node-ip>.nip.io. Some network policies block these names; use corporate DNS or an approved /etc/hosts mapping in that case.
Enable hybrid mode
  1. Change hybrid.enable from the template default false to true.
  2. Change the values as required.
    Configuration key Value to supply
    hybrid.controller.domainName Standalone Controller or load-balancer DNS name
    hybrid.controller.port Standalone Controller port; template default is 8181
    hybrid.controller.sslEnabled true when the Controller endpoint uses TLS
    hybrid.mysql.dbHost Standalone Controller database or load-balancer DNS name
    hybrid.mysql.dbPort Controller database port; template default is 3388
    hybrid.kafka.defaultCert true for the generated default certificate; use the release-approved setting when supplying custom Kafka certificate files
    hybrid.schemaregistry.externalUrl https://<VA_DNS_DOMAIN>/schemaregistry
    Warning: Do not manually replace the Kafka range loop. It generates kafkaHost0…n from the detected VA node IPs and assigns ports beginning at 32101.

    In high-availability, without a load balancer or virtual IP, update hybrid.controller.domainName and hybrid.mysql.dbHost after a Controller failover. With a load balancer, use its DNS name and applicable ports.

  3. To configure secrets, edit /var/appd/config/secrets.yaml before the first installation. After installation, the file is encrypted as secrets.yaml.encrypted
    Update the required passwords in this section.
    YAML 
    hybrid:
	controller:
		controllerKey: <Controller-Key-Value>
		tenantAccountName: customer1
		rootUsername: root
		rootPassword: welcome
		rootAccountname: system
		adminUsername: admin
		adminPassword: welcome
	mysql:
		dbUser: secapp
		dbPassword: changeit
  4. Run the supplied DNS checker.
    CODE 
    bash /home/appduser/appd-charts/utils/check-dns.sh
    Sample Output:
    CODE 
    Deployment mode: hybrid
Virtual Appliance tenant: 'e2e-customer'
Virtual Appliance domain: 'bf-manual-va.e2e.appd-test.com'
Controller tenant: 'e2e-customer'
Controller domain: 'bf-manual-cont-es-ec-controller.e2e.appd-test.com'
[OK] bf-manual-cont-es-ec-controller.e2e.appd-test.com
[OK] bf-manual-va.e2e.appd-test.com
[OK] e2e-customer.bf-manual-cont-es-ec-controller.e2e.appd-test.com
[OK] e2e-customer-tnt-con.bf-manual-va.e2e.appd-test.com
[OK] e2e-customer-tnt-con.bf-manual-cont-es-ec-controller.e2e.appd-test.com
[OK] localhost
[OK] bf-manual-va.e2e.appd-test.com
[OK] 10.115.84.215.nip.io
[OK] 10.115.85.239.nip.io
[OK] 10.115.87.4.nip.io
[OK] e2e-customer.auth.bf-manual-va.e2e.appd-test.com
[OK] e2e-customer-tnt-authn.bf-manual-va.e2e.appd-test.com

DNS verification completed: 0 unresolved host(s)
    The checker ends with DNS verification completed: 0 unresolved host(s). The standalone Controller must also resolve the Virtual Appliance authentication names to the Virtual Appliance ingress IP.
  5. Optional: Replace the Virtual Appliance Ingress self-signed certificates.
    1. Generate the required SAN list and.
      CODE 
      bash /home/appduser/appd-charts/utils/list-ingress-sans.sh
      Sample Output:
      CODE 
      Deployment mode: hybrid
Virtual Appliance tenant: 'e2e-customer'
Virtual Appliance domain: 'bf-manual-va.e2e.appd-test.com'
Virtual Appliance node names: localhost bf-manual-va.e2e.appd-test.com 10.115.84.215.nip.io 10.115.85.239.nip.io 10.115.87.4.nip.io

Include these Subject Alternative Names in the custom ingress certificate:
  bf-manual-va.e2e.appd-test.com
  e2e-customer.bf-manual-va.e2e.appd-test.com
  *.bf-manual-va.e2e.appd-test.com
  e2e-customer.auth.bf-manual-va.e2e.appd-test.com
  e2e-customer-tnt-authn.bf-manual-va.e2e.appd-test.com
  localhost
  10.115.84.215.nip.io
  10.115.85.239.nip.io
  10.115.87.4.nip.io
    2. Obtain a certificate whose SANs include every configured DNS name.
    3. Copy the PEM key and ordered certificate bundle to the primary node.
      Copy the CA provided ingress.crt and ingress.key to the following location:
      CODE 
      /var/appd/config folder of primary node
    Note:
    • ingress.crt must contain the leaf/server certificate, intermediate CA certificate(s), then root CA certificate.
    • If a load balancer terminates TLS, retrieve and trust the certificate chain presented by the load balancer, not the backend Controller.
    • For a custom Kafka certificate, include all three VA node IP addresses in its SANs.
  6. Copy the Controller license file to /var/appd/config, then allow the VA Kafka node ports on every cluster node:
    CODE 
    sudo ufw allow AppdNodePorts

Collect Certificates to Establish TLS Trust

Establish TLS trust in the following distinct paths in Virtual Appliance:
Trust Path Action on Virtual Appliance Expected files
Virtual Appliance to Controller or its load balancer Retrieve the certificate chain presented by the endpoint.
CODE 
bash /home/appduser/appd-charts/utils/get-hybrid-controller-ca.sh
 /var/appd/config/hybrid-controller-ca.crt
Virtual Appliance to MySQL Retrieve the database certificate chain. Skip only if MySQL does not use TLS.Certificate file.
CODE 
bash /home/appduser/appd-charts/utils/get-hybrid-mysql-ca.sh
 /var/appd/config/hybrid-mysql-ca.crt
Controller to Kafka on Virtual Appliance Keep Virtual Appliance's self-signed certificates, or provide a CA-signed key and certificate. /var/appd/config/hybrid-kafka.key and
  1. Retrieve the trust chains presented by Controller and MySQL.
    CODE 
    # Retrieve trust chains already presented by the Controller and MySQL
bash /home/appduser/appd-charts/utils/get-hybrid-controller-ca.sh
    Sample Output:
    PYTHON 
    bash /home/appduser/appd-charts/utils/get-hybrid-controller-ca.sh https://bf-manual-cont-es-ec-controller.e2e.appd-test.com:8181
Retrieving the TLS certificate chain from bf-manual-cont-es-ec-controller.e2e.appd-test.com:8181...
Saved 1 Controller CA certificate(s) to /var/appd/config/hybrid-controller-ca.crt
subject=CN = bf-manual-va.e2e.appd-test.com
issuer=CN = bf-manual-va.e2e.appd-test.com

Kubernetes Secret cisco-controller/controller-tls-secret does not exist; Helm will create it from /var/appd/config/hybrid-controller-ca.crt.
  2. Retrieve the database certificate chain.
    CODE 
    bash /home/appduser/appd-charts/utils/get-hybrid-mysql-ca.sh

# Confirm files were created
ls -l /var/appd/config/hybrid-controller-ca.crt \
      /var/appd/config/hybrid-mysql-ca.crt
    Note: If the MySQL certificates are missing in the classic Splunk AppDynamics On-Premises environment, the script cannot retrieve the MySQL trust chain and will display a warning. You may safely ignore this warning and proceed to the next step.