Install Cisco AppDynamics Services in the Hybrid Deployment

In a hybrid deployment, Splunk AppDynamics On-Premises Virtual Appliance leverages your current Controller, Events Service, and End User Monitoring components from Splunk AppDynamics On-Premises and installs Anomaly Detection and Cisco Secure Application Services in your Kubernetes cluster.

Prepare to Install Splunk AppDynamics Services

Complete the following steps to prepare the environment:

Log into the console of the primary node using the appduser credentials.This node is considered as the primary node after you create a three-node cluster.
Navigate to the following folder:
```
cd /var/appd/config
```

Edit the globals.yaml.gotmpl file with the required configuration.

vi globals.yaml.gotmpl

Specify the Controller key and the account name.
Enable the hybrid mode.
Set the Controller domain name.
Configure the port to access the standalone Controller.
Set the sslEnabled field to true .
Upload the Controller CA certificates for the standalone Controller.
Configure the MySQL host. It is the same host as the Controller domain.
Configure a port for the standalone Controller database.
Update the MySQL CA certificates, which are located in <controller-home>/platform/product/controller/db/data/ca.pem.
(Optional) Update the certificate and key of the Kafka service in the hybrid.kafka section of globals.yaml.gotmpl.

(Optional) Configure a custom ingress certificate (by default the ingress controller installs a fully-configured self-signed certificate). The custom ingress certificate needs certain SANs added to it. Run the following script on the console of your primary node to view a list of SANs. Add those SANs to your custom ingress certificate. See ingress in Customize the Helm File for instructions on how to configure the custom ingress certificate and key.

#!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
DNS_NAMES=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsNames|join(" ")')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Virtual Appliance node names are: ${DNS_NAMES}

echo If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
for server_name in "$DNS_DOMAIN" "${TENANT}.${DNS_DOMAIN}" "*.${DNS_DOMAIN}" "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}" $DNS_NAMES; do
        echo "  ${server_name}"
done

Sample output of the script above:

Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Virtual Appliance node names are: localhost vanodename-1
If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
  va.mycompany.com
  customer1.va.mycompany.com
  *.va.mycompany.com
  customer1.auth.va.mycompany.com
  customer1-tnt-authn.va.mycompany.com
  localhost
  vanodename-1

Configure secrets in the Virtual Appliance:

Update the Controller and MySQL credentials.

hybrid:
  controller:
    controllerKey: <your-controller-access-key>
    tenantAccountName: <tenant-name>
    rootUsername: root
    rootPassword: '<root-password>'
    rootAccountname: system
    adminUsername: admin
    adminPassword: '<admin-password>'
  mysql:
    dbUser: secapp
    dbPassword: '<string>'

Edit the usernames and passwords of the Splunk AppDynamics services.

Update the firewall rules to allow the standalone Controller to access the Kafka ports in the Kubernetes cluster. Update the firewall rule in each cluster node.
```
sudo ufw allow AppdNodePorts
```

Create a Three-Node Cluster

Note: Skip creating a three-node cluster for the demo profile.

Log in to the primary node console.
Verify the boot status of each node of the cluster:
```
appdctl show boot
```
Note:
- Ensure the status of the services in each node appears as Success. Else, restart the virtual machine that failed.
  If it is still failing, you might have to redeploy the virtual machine.
- Ensure to configure the same time on all the cluster nodes.
Run the following command in the primary node and specify the IP address of the peer nodes:
```
cd /home/appduser
appdctl cluster init <Node-2-IP> <Node-3-IP>
```

Run the following command to verify the node status:

appdctl show cluster
microk8s status

Ensure that the output displays the Running status as true for the nodes that are part of the cluster.

Sample Output

 NODE           | ROLE  | RUNNING 
----------------+-------+---------
 10.0.0.1:19001 | voter | true    
 10.0.0.2:19001 | voter | true    
 10.0.0.3:19001 | voter | true

Note: You must re-login to the terminal if the following error appears:

Insufficient Permissions to Access Microk8s

Configure the Standalone Controller in Splunk AppDynamics On-Premises

Log in to the Controller database by using the following command:
```
<controller-home>/bin/controller.sh login-db
```
Connect to the MySQL database.
Run this command on the Virtual Appliance primary node to get the value of dbUser:
```
helm secrets decrypt /var/appd/config/secrets.yaml.encrypted | yq .hybrid.mysql.dbUser
```
Note: The value of hybrid.mysql.dbUser is specific to Cisco Secure Application and does not ever need to be changed from its default value of secureapp.

Run this command on the Virtual Appliance primary node to get the value of dbPassword:

helm secrets decrypt /var/appd/config/secrets.yaml.encrypted | yq .hybrid.mysql.dbPassword

Drop and create the <dbUser> for Cisco Secure Application Services and grant permissions to access the Controller database, replacing <dbUser> and <dbPassword> with the values you retrieved:

DROP USER <dbUser>;
CREATE USER '<dbUser>'@'%' identified by '<dbPassword>' REQUIRE SSL;
GRANT USAGE ON *.* TO `<dbUser>`@`%`;
GRANT SELECT  ON `controller`.* TO `<dbUser>`@`%`;  
GRANT UPDATE  ON `controller`.* TO `<dbUser>`@`%`;

Access the Enterprise Console UI.
Select the platform.
Go to the Configuration tab.
Navigate to Controller Settings > Appserver Configurations > JVM Options.

Replace <DNS_DOMAIN> with the value configured as dnsDomain in /var/appd/config/globals.yaml.gotmpl:

-Dappdynamics.auth.service.hosted.zone=-tnt-authn.<DNS_DOMAIN>
-Dappdynamics.auth.service.port=443
-Dappdynamics.auth.service.ssl.access.enabled=true
-Dappdynamics.auth.service.token.generation.enabled=true
-Dappdynamics.controller.authentication.resource.accessible=true
-Dappdynamics.auth.client.token.validation.enabled=true

Note:

Alternatively, add these authentication settings in <controller-home>/platform/product/controller/appserver/jetty/start.d/start.ini

Note:

Changes won't take effect until you restart the standalone Controller in a later step.

Install Services in the Cluster

Run the command to install services:

Demo Profile

appdcli start appd demo

Small Profile

appdcli start appd small

Medium Profile

appdcli start appd medium

Large Profile

appdcli start appd large

This command installs the Splunk AppDynamics services. We recommend you to specify the VA profile as same as the profile that you selected to create a virtual machine. See, Virtual Appliance Sizing.

Sample Output

NAME               CHART                     VERSION   DURATION
cert-manager-ext   charts/cert-manager-ext   0.0.1           0s
ingress-nginx      charts/ingress-nginx      4.8.3           1s
redis-ext          charts/redis-ext          0.0.1           1s
ingress            charts/ingress            0.0.1           2s
cluster            charts/cluster            0.0.1           2s
reflector          charts/reflector          7.1.216         2s
monitoring-ext     charts/monitoring-ext     0.0.1           2s
minio-ext          charts/minio-ext          0.0.1           2s
eum                charts/eum                0.0.1           2s
fluent-bit         charts/fluent-bit         0.39.0          2s
postgres           charts/postgres           0.0.1           2s
mysql              charts/mysql              0.0.1           3s
redis              charts/redis              18.1.6          3s
controller         charts/controller         0.0.1           3s
events             charts/events             0.0.1           4s
cluster-agent      charts/cluster-agent      1.16.37         4s
kafka              charts/kafka              0.0.1           6s
minio              charts/minio              5.0.14         47s

Verify the status of the installed pods and service endpoints:
- Pods: kubectl get pods --all-namespaces

Note:

When a Virtual Machine restarts, the MySQL service may not automatically restore. To troubleshoot this issue, see Restore the MySQL Service.
If the EUM Collector, EUM Aggregator, and EUM Screenshot pods are failing even after multiple retries, see EUM Health is Failing After Multiple Retries.

ハイブリッド展開の DNS エントリの設定

プライマリ仮想アプライアンスノードのコンソールに、次のスクリプトを dnsinfo.sh として保存し、実行します。その場合は、出力の手順に従ってください。

#!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
CONTROLLER_DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.hybrid.controller.domainName')
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Controller domain name should be \'${CONTROLLER_DNS_DOMAIN}\'
 
echo
for server_name in "$CONTROLLER_DNS_DOMAIN" "${DNS_DOMAIN}" "${TENANT}.${CONTROLLER_DNS_DOMAIN}" "${TENANT}-tnt-con.${DNS_DOMAIN}" "${TENANT}-tnt-con.${CONTROLLER_DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please add DNS entry for ${server_name} for controller host IP, VA is not able to resolve it currently"
  fi
done
for server_name in "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}"; do
  if ! getent hosts "${server_name}" > /dev/null; then
    echo "Please double-check on standalone controller that DNS can resolve entry for ${server_name} as VA ingress IP"
  fi
done

スクリプトの呼び出し例：

bash ./dnsinfo.sh

サンプル出力：

Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Controller domain name should be 'controller.mycompany.com'

Please add DNS entry for controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for va.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1.controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1-tnt-con.va.mycompany.com for controller host IP, VA is not able to resolve it currently
Please add DNS entry for customer1-tnt-con.controller.mycompany.com for controller host IP, VA is not able to resolve it currently
Please double-check on standalone controller that DNS can resolve entry for customer1.auth.va.mycompany.com as VA ingress IP
Please double-check on standalone controller that DNS can resolve entry for customer1-tnt-authn.va.mycompany.com as VA ingress IP

Update the Standalone Controller TLS Certificate

Note: For more information, see Controller SSL and Certificates.

Note: Changes won't take effect until you restart the standalone Controller in a later step.

Create a new standalone Controller certificate.Save the following script on the console of your primary virtual appliance node as certinfo.sh and run it. Follow the instructions in its output:

#!/bin/bash
set -euo pipefail
TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
CONTROLLER_DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.hybrid.controller.domainName')
DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
echo Verify the Controller domain name should be \'${CONTROLLER_DNS_DOMAIN}\'
 
echo When creating and importing the Controller Server Certificate, include the following SANs:
for server_name in "${CONTROLLER_DNS_DOMAIN}" "${DNS_DOMAIN}" "${TENANT}.${CONTROLLER_DNS_DOMAIN}" "${TENANT}-tnt-con.${DNS_DOMAIN}" "${TENANT}-tnt-con.${CONTROLLER_DNS_DOMAIN}"; do
  echo "  ${server_name}"
done

Sample script invocation:

bash ./certinfo.sh

Sample output:

Verify the Virtual Appliance tenant should be 'customer1'
Verify the Virtual Appliance domain name should be 'va.mycompany.com'
Verify the Controller domain name should be 'controller.mycompany.com'
When creating and importing the Controller Server Certificate, include the following SANs:
  controller.mycompany.com
  va.mycompany.com
  customer1.controller.mycompany.com
  customer1-tnt-con.va.mycompany.com
  customer1-tnt-con.controller.mycompany.com

Import the newly created certificate that has the DNS entries you added:

keytool -import -trustcacerts -alias s1as \
-file <certificate.crt> \
-keystore <ControllerInstallation>/platform/product/controller/appserver/jetty/etc/keystore.jks

Generate the Hybrid Configuration File

Note: If you skipped updating the CA certificates, you must regenerate hybrid configuration file after restarting a Virtual Appliance service.

Run the script to connect the Kafka node ports with the standalone controller:

appduser@appd-demo-2:~/appd-charts/utils$ bash prepare-hybrid.sh

Sample Output

Writing Kafka CA Cert
Writing Schema registry CA Cert
Writing AnomalyDetectionKafkaSSLProducer
Writing AnomalyDetectionKafkaConsumerSSL
Writing configure script
Creating archive of hybrid config
hybrid-config/
hybrid-config/kafka-ca.crt
hybrid-config/AnomalyDetectionKafkaConsumerSSL
hybrid-config/configure.sh
hybrid-config/schema-registry-ca.crt
hybrid-config/AnomalyDetectionKafkaSSLProducer
Completed

This script generates the hybrid-config.tar file that includes the required configuration files.

Copy the hybrid-config.tar file to your standalone controller instance.
Locate and extract the hybrid-config.tar file contents.

Configure the Controller Properties by Using the Kubernetes Cluster CLI

Note:

Changes won't take effect until you restart the standalone Controller in a later step.
If you skipped updating the CA certificates, you must regenerate hybrid configuration file after restarting a Virtual Appliance service.

Run the command to update the standalone Controller properties:

appdcli run bootstrap_hybrid

This command configures the following properties in the Controller:

Key: anomaly.detection.enabled, Value: true
Key: anomaly.detection.deployed.onprem, Value: true
Key: controller.id, Value: onprem
Key: controller.unique.id, Value: onprem
Key: anomaly.detection.kafka.consumer.group.id, Value: pi-alarmservice-transition-consumer-group-onprem
Key: anomaly.detection.kafka.consumer.topic, Value: pi-alarmservice-transition-onprem
Key: anomaly.detection.kafka.producer.topic, Value: metadata-lifecycle-events
Key: anomaly.detection.onprem.output.topic, Value: blitz-processed-metrics
Key: anomaly.detection.schema.registry.enabled, Value: true
Key: anomaly.detection.consumer.schema.registry.url, Value: https://10.0.0.1.nip.io/schemaregistry
Key: anomaly.detection.producer.schema.registry.url, Value: https://10.0.0.2.nip.io/schemaregistry
Key: anomaly.detection.kafka.consumer.bootstrap.servers, Value: 10.0.0.1:32101
Key: anomaly.detection.kafka.producer.bootstrap.servers, Value: 10.0.0.2:32102
Key: anomaly.detection.kafka.producer.ssl.enabled, Value: true
Key: anomaly.detection.kafka.consumer.ssl.enabled, Value: true
Key: anomaly.detection.service.url, Value: https://10.0.0.1.nip.io/pi

Alternatively, you can access the administration console by using the Kubernetes cluster IP address and manually edit each parameter.

Run the configure.sh script to provision the controller with certs and secrets to connect to Kubernetes cluster:

This script is extracted from the hybrid-config.tar file. See how to obtain hybrid-config.tar file.

$ bash configure.sh
Usage: configure.sh <controller-home> <keystore-passwd>
Using controller home as /opt/appdynamics and keystore password as changeit
Do you want to proceed? (y/n): y
Creating Kafka truststore
Certificate was added to keystore
Creating schema registry truststore
Certificate was added to keystore
Writing obfuscated producer config
Writing obfuscated consumer config
Completed

スタンドアロンコントローラを再起動します。

次のコマンドを実行して、スタンドアロンコントローラを再起動します。

注: このコマンドは MySQL サービスも再起動します。

cd ~/appdynamics/platform/product/controller/bin/
./controller.sh stop
./controller.sh start

Install the Anomaly Detection Services in the Cluster

Run the command to install services:

Demo Profile

appdcli start aiops demo

Small Profile

appdcli start aiops small

Medium Profile

appdcli start aiops medium

Large Profile

appdcli start aiops large

Verify the status of the installed pods and service endpoints:
- Pods: kubectl get pods --all-namespaces
- Service endpoints: appdcli ping

See Anomaly Detection.

Note: Sometimes, IOException error occurs when you access Anomaly Detection in the Controller UI. See Troubleshoot Virtual Appliance Issues.

（オプション）Cisco Secure Application をインストールします

Install Cisco Secure Application を参照してください。

クラスターへの ATD サービスのインストール

AuthN サービスが Splunk AppDynamics サービスとともにインストールされていることを確認します。

kubectl get pods -nauthn

仮想アプライアンスに自動トランザクション診断サービスをインストールする手順は次のとおりです。

クラスターノードコンソールにログインします。
コマンドを実行して、サービスをインストールします。
デモプロファイル
```
appdcli start atd demo
```
小規模プロファイル
```
appdcli start atd small
```
中規模プロファイル
```
appdcli start atd medium
```
インストールされたポッドとサービスエンドポイントのステータスを確認します。
```
kubectl get pods -ncisco-atd
```
ATD の詳細については、「自動トランザクション診断のワークフロー」を参照してください。

Download Splunk AppDynamics Agents

Splunk AppDynamics On-Premises Virtual Appliance supports you to install the Splunk AppDynamics agents. Download the agents from Download Portal.

For more information, see: