Install Services in the Cluster

Run the command to install services:

Small Profile

appdcli start appd small

Medium Profile

appdcli start appd medium

Large Profile

appdcli start appd large

This command installs the Splunk AppDynamics services. We recommend you to specify the VA profile as same as the profile that you selected to create a virtual machine. See, Virtual Appliance Sizing.

Sample Output

NAME               CHART                     VERSION   DURATION
cert-manager-ext   charts/cert-manager-ext   0.0.1           0s
ingress-nginx      charts/ingress-nginx      4.8.3           1s
redis-ext          charts/redis-ext          0.0.1           1s
ingress            charts/ingress            0.0.1           2s
cluster            charts/cluster            0.0.1           2s
reflector          charts/reflector          7.1.216         2s
monitoring-ext     charts/monitoring-ext     0.0.1           2s
minio-ext          charts/minio-ext          0.0.1           2s
eum                charts/eum                0.0.1           2s
fluent-bit         charts/fluent-bit         0.39.0          2s
postgres           charts/postgres           0.0.1           2s
mysql              charts/mysql              0.0.1           3s
redis              charts/redis              18.1.6          3s
controller         charts/controller         0.0.1           3s
events             charts/events             0.0.1           4s
cluster-agent      charts/cluster-agent      1.16.37         4s
kafka              charts/kafka              0.0.1           6s
minio              charts/minio              5.0.14         47s

Verify the status of the installed pods and service endpoints:
Pods: kubectl get pods --all-namespaces
Note: Wait until all pods are up and running.
You can also access the endpoint URL to verify the installation. See Verify the Service Endpoints Paths.

Note: When a Virtual Machine restarts, the MySQL service may not automatically restore. To troubleshoot this issue, see Restore the MySQL Service.

Configure the Standalone Controller in Splunk AppDynamics On-Premises

Log into the Controller database by using the following command:
```
<controller-home>/bin/controller.sh login-db
```
Note: You must enter the root database password to connect to the MySQL database.
Run this command on the Virtual Appliance primary node to get the value of dbUser:
```
helm secrets decrypt /var/appd/config/secrets.yaml.encrypted | yq .hybrid.mysql.dbUser
```
Note: The hybrid.mysql.dbUser value is for Secure Application only. Do not change it from the default value: secureapp.

Run this command on the Virtual Appliance primary node to get the value of dbPassword:

helm secrets decrypt /var/appd/config/secrets.yaml.encrypted | yq .hybrid.mysql.dbPassword

Using the value of <dbUser> from step 3 (which should be secureapp), drop and create <dbUser> for Secure Application Services and grant permissions to access the Controller database, replacing <dbUser> and <dbPassword> with the values you retrieved:
```
DROP USER <dbUser>;
CREATE USER '<dbUser>'@'%' identified by '<dbPassword>' REQUIRE SSL;
GRANT USAGE ON *.* TO `<dbUser>`@`%`;
GRANT SELECT  ON `controller`.* TO `<dbUser>`@`%`;  
GRANT UPDATE  ON `controller`.* TO `<dbUser>`@`%`; 
```
(Optional) Generate the hybrid configuration file to prevent multiple Controller restarts.

Access the Enterprise Console UI.

Select the platform.
Go to the Configuration tab.
Navigate to Controller Settings > Appserver Configurations > JVM Options > JVM Config.

In the JVM Config field, append the following content:

Note: Replace <DNS_DOMAIN> with the value that you configured as dnsDomain in /var/appd/config/globals.yaml.gotmpl.

-Dappdynamics.auth.service.hosted.zone=-tnt-authn.<DNS_DOMAIN>
-Dappdynamics.auth.service.port=443
-Dappdynamics.auth.service.ssl.access.enabled=true
-Dappdynamics.auth.service.token.generation.enabled=true
-Dappdynamics.controller.authentication.resource.accessible=true
-Dappdynamics.auth.client.token.validation.enabled=true
-Dappdynamics.mds.rbac.rest.api.enabled=true
-Dappdynamics.rbac.internal.resource=true

Click Save.
Note: After you save the configuration, the Controller restarts.

Install the Anomaly Detection Services in the Cluster

Log into the cluster node console.
Run the command to install services:
Small Profile
```
appdcli start aiops small
```
Medium Profile
```
appdcli start aiops medium
```
Verify the status of the installed pods and service endpoints:
- Pods: kubectl get pods --all-namespaces
- Service endpoints: appdcli ping

See Anomaly Detection.

Note: Sometimes, an IOException error occurs when you access Anomaly Detection in the Controller UI. See Troubleshoot Virtual Appliance Issues.

Install ATD Services

Ensure that:

DNS entries are added that are required for the resolution of the ATD services. Else, configure the DNS entries using the encrypted secrets file. See Configure DNS Entries.
Ingress certificate and key with all necessary Subject Alternative Names (SANs) are generated and updated the globals.yaml.gotmpl file accordingly. Configure Ingress certificates using the encrypted secrets file. See (Optional) Configure Ingress Certificates.
AuthN service is installed along with Splunk AppDynamics services.
```
kubectl get pods -nauthn
```

Follow the steps to install Automatic Transaction Diagnostics service in the Virtual Appliance:

Run the command to install services:

Demo Profile

appdcli start atd demo

Small Profile

appdcli start atd small

Medium Profile

appdcli start atd medium

Large Profile

appdcli start atd large

Verify the status of installed pods and service endpoints.
```
kubectl get pods -ncisco-atd
```
You can also access the endpoint URL to verify the installation. See Verify the Service Endpoints Paths.
For more information about ATD, see Automated Transaction Diagnostics Workflow.

Install OpenTelemetry Service

Ensure the following conditions are met:

Controller is 24.10 or higher on your On-Premises Classic deployment. See Upgrade the Controller Using Enterprise Console
OpenTelemetry Collector version 0.36 to 0.101.
The maximum size limit for each request sent to the trace ingestion endpoint is 10 MB.
DNS entries are added that are required for the resolution of the OpenTelemetry services. Else, configure the DNS entries using the encrypted secrets file. See Configure DNS Entries.
Generated the Ingress certificate and key with all necessary Subject Alternative Names (SANs) and updated the globals.yaml.gotmpl file accordingly. Configure Ingress certificates using the encrypted secrets file. See (Optional) Configure Ingress Certificates.

Go to the Controller DNS to verify that the Controller is active.
Log in to the cluster node console.
Run the following command to install the OpenTelemetry™ service:
Small Profile
```
appdcli start otis small
```
Medium Profile
```
appdcli start otis medium
```
Large Profile
```
appdcli start otis large
```
This command installs the OpenTelemetry™ service in the cisco-otis namespace.

Verify the status of the installed pods and service endpoints:

Pods: kubectl get pods -n cisco-otis

Service endpoints: appdcli ping

The status of the OpenTelemetry™ service appears as Success.

+---------------------+---------------+
|  Service Endpoint   |    Status     |
+=====================+===============+
| Controller          | Success       |
+---------------------+---------------+
| Events              | Success       |
+---------------------+---------------+
| EUM Collector       | Success       |
+---------------------+---------------+
| EUM Aggregator      | Success       |
+---------------------+---------------+
| EUM Screenshot      | Success       |
+---------------------+---------------+
| Synthetic Shepherd  | Success       |
+---------------------+---------------+
| Synthetic Scheduler | Success       |
+---------------------+---------------+
| Synthetic Feeder    | Success       |
+---------------------+---------------+
| OTIS                | Success       |
+---------------------+---------------+

You can also access the endpoint URL to verify the installation. See Verify the Service Endpoints Paths.

Configure Splunk AppDynamics for OpenTelemetry.

(Optional) Install Secure Application

Ensure:

DNS entries are added that are required for the resolution of the Secure Application services. Else, configure the DNS entries using the encrypted secrets file. See Configure DNS Entries.
Generated the Ingress certificate and key with all necessary Subject Alternative Names (SANs) and updated the globals.yaml.gotmpl file accordingly. Configure Ingress certificates using the encrypted secrets file. See (Optional) Configure Ingress Certificates.

See Install Secure Application.

Install Universal Integration Layer Service

Ensure that:

DNS entries are added that are required for the resolution of the Anomaly services. Else, configure the DNS entries using the encrypted secrets file. See Configure DNS Entries.
The Ingress certificate and key with all necessary Subject Alternative Names (SANs) are generated and the globals.yaml.gotmpl file is updated accordingly. Configure Ingress certificates using the encrypted secrets file. See (Optional) Configure Ingress Certificates.
The authentication settings to the Controller configuration are added. See Configure Standalone Controller.

To integrate Splunk AppDynamics Self Hosted Virtual Appliance with Splunk Enterprise, you must install the Universal Integration Layer (UIL) service in the cluster:

Log in to the cluster node console.
Run the command to install the service:
Small Profile
```
appdcli start uil small
```
Medium Profile
```
appdcli start uil medium
```
Large Profile
```
appdcli start uil large
```
Verify the status of the installed pods and service endpoints:
- Pods: kubectl get pods -n cisco-uil
  The status of the universal integration layer pod must be displayed as Running.
- Service endpoints: appdcli ping
  The status of the UIL service should be displayed as Success.
You can also access the endpoint URL to verify the installation. See Verify the Service Endpoints Paths.

To continue with the integration, see Integrate Splunk AppDynamics Self Hosted Virtual Appliance with Splunk Enterprise.