Set up PSA in Bare Metal K8s Using Amazon EC2

Set up the Web Monitoring PSA and API Monitoring PSA in Bare Metal K8s using Amazon EC2 as follows. If you want to set up PSA in an existing Kubernetes cluster, skip the Create the Kubernetes Cluster section.

Create the Kubernetes Cluster.
Pull the Docker image.
Save Images to Minikube's Docker Daemon.
Deploy the Web Monitoring PSA and API Monitoring PSA.
Monitor the Kubernetes cluster.

Note: Most of the steps are common for both Web Monitoring PSA and API Monitoring PSA. Wherever applicable, the differences in steps are highlighted.

Warning: This document contains links to AWS CLI documentation. Splunk AppDynamics makes no representation as to the accuracy of AWS CLI documentation because AWS CLI controls its own documentation.

Create the Kubernetes Cluster

Note: You can use kops to create your own managed Kubernetes cluster on AWS. If you want to create self-managed cluster on a different cloud or your own datacenter, you might want to look into other tools like Kubeadm or Kubespray. See installing Kubernetes with deployment tools.

To create a Kubernetes cluster in Bare Metal K8s:

Install and configure AWS CLI.

To create IAM Role, enter.

aws iam create-group --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops
aws iam create-user --user-name kops
aws iam add-user-to-group --user-name kops --group-name kops
aws iam create-access-key --user-name kops

Configure AWS CLI credentials using aws configure with the access key obtained in the previous step.
Based on your platform, install kops following these instructions.

To create S3 bucket, enable versioning by entering:

bucket_name=heimdall-onprem-kops-state-store
aws s3api create-bucket \
--bucket ${bucket_name} \
--create-bucket-configuration LocationConstraint=us-west-2
aws s3api put-bucket-versioning --bucket ${bucket_name} --versioning-configuration Status=Enabled

To create cluster, enter:

export KOPS_CLUSTER_NAME=heimdall-onprem.k8s.local
export KOPS_STATE_STORE=s3://${bucket_name}
export KOPS_KUBERNETES_VERSION=1.x.x
kops create cluster \
--node-count=4 \
--node-size=t3.2xlarge \
--zones=us-west-2a \
--kubernetes-version=${KOPS_KUBERNETES_VERSION} \
--name=${KOPS_CLUSTER_NAME}
kops update cluster --name ${KOPS_CLUSTER_NAME} --yes

Note: Specify the KOPS_KUBERNETES_VERSION from one of the compatible versions.

Note: The node-size and node-count in the above code snippet are selected according to recommended configuration type. You can specify a configuration of your choice with a different type and number of nodes. See EC2 instance types.

To validate if the cluster is running (might take some time for cluster to set up and run), enter:
```
kops validate cluster
```

Access the Cluster

To access the Kubernetes cluster, follow these instructions to install kubectl, a utility to interact with the cluster.

To verify that the cluster is running, enter:

kubectl get nodes

(Optional) Configure Proxy Server

When you configure a proxy server, it applies to all the domains. Configure a proxy server by specifying the proxy server address on the values.yaml file. See Key-Value Pairs Configuration.

To bypass any domains from the proxy server, perform the following steps:

Note: Configuring the bypass list is supported only on Web Monitoring PSA.

Open the values.yaml file.
Add the domain URLs in the bypassList field under browserMonitoringAgent:
```
browserMonitoringAgent:
enabled: true
server: "<proxy server address>"
bypassList: "<specify the domain URLs that you want to bypass separated by semicolon>"
```
For example, bypassList: "*abc.com;*xyz1.com;*xyz2.com"
Domain URLs that you specify in bypassList are not redirected to the proxy server. You can add any number of domains in the bypassList. All other unspecified domain URLs are redirected to the proxy server.

Configure Proxy Server at a Job Level

You can configure a proxy server and a bypass list for a particular job. Any proxy server URL and bypass list configured at the agent level gets overridden by the proxy server URL and bypass list configured at the job level. Perform the following steps to configure a proxy server at the job level:

Create or edit a Synthetic job.
Select the Run a script option.

Add the following details in the beginning the script:

'''yml
jobLevelProxyConfig:
  proxyServer: "<proxy-server-address>"
  bypassProxyList: "<list of URLs that you want to bypass, separated by semi-colon>"
'''

Example:

'''yml
jobLevelProxyConfig：
  proxyServer: "http://tinyproxy: tinyproxy.svc.cluster.local:8888"
  bypassProxyList: "*abc.com;*xyz1.com;*xyz2.com"
'''

pageUrl = "https://help.splunk.com/en"
driver.get(pageUrl)
assert "Splunk" in driver.title, "Title should contain Splunk"

Pull the Docker Image

Pull the pre-built docker images for sum-chrome-agent, sum-api-monitoring-agent, and sum-heimdall from DockerHub. The pre-built images include the dependent libraries, so you can use these images even when you do not have access to the Internet.

Run the following commands to pull the agent images:

docker pull appdynamics/heimdall-psa
docker pull appdynamics/chrome-agent-psa
docker pull appdynamics/api-monitoring-agent-psa

(Optional) Add Custom Python Libraries

Note: This section is applicable only for Web Monitoring PSA.

In addition to the available standard set of libraries, you can add custom Python libraries to the agent to use in scripted measurements. You build a new image based on the image you loaded as the base image

Create a Dockerfile and create RUN directives to run python pip. For example, to install the library algorithms you can create a Dockerfile:

# Use the sum-chrome-agent image you just loaded as the base image
FROM appdynamics/chrome-agent-psa:<agent-tag>
USER root
RUN apk add py3-pip
USER appdynamics
# Install algorithm for python3 on top of that
RUN python3 -m pip install algorithms==0.1.4 --break-system-packages

Note: You can create any number of RUN directives to install the required libraries.

To build the new image, enter:
```
docker build -t sum-chrome-agent:<agent-tag> - < Dockerfile
```
The newly built agent image contains the required libraries.

Tag and Push Images to the Registry

You must tag and push the images to a registry for the cluster to access it. It can be done in the following ways:

Bare Metal K8S using EC2

Vanilla K8S runs on AWS infrastructure. As kops create and assign appropriate roles to the cluster nodes, they can directly access Elastic Container Registry (ECR), without any other configuration.

To tag images, enter:

Web Monitoring PSA:

docker tag sum-heimdall:<heimdall-tag> <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-heimdall:<heimdall-tag>
docker tag sum-chrome-agent:<agent-tag> <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-chrome-agent:<agent-tag>

API Monitoring PSA:

docker tag sum-heimdall:<heimdall-tag> <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-heimdall:<heimdall-tag>
docker tag sum-api-monitoring-agent:<agent-tag> <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-api-monitoring-agent:<agent-tag>

Replace <aws_account_id> and <region> with your account and region values.

To create repositories, enter:

Web Monitoring PSA:

aws ecr create-repository --repository-name sum/sum-heimdall
aws ecr create-repository --repository-name sum/sum-chrome-agent

API Monitoring PSA:

aws ecr create-repository --repository-name sum/sum-heimdall
aws ecr create-repository --repository-name sum/sum-api-monitoring-agent

To push the images, enter:

Web Monitoring PSA:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
docker push <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-heimdall:<heimdall-tag>
docker push <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-chrome-agent:<agent-tag>

API Monitoring PSA:

aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
docker push <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-heimdall:<heimdall-tag>
docker push <aws_account_id>.dkr.ecr.<region>.amazonaws.com/sum/sum-api-monitoring-agent:<agent-tag>

Bare Metal K8S using Private Registry

Note: If you are managing your own Kubernetes cluster which is not on AWS, then you must deploy your own registry server. See deploy a registry server. See deploy a registry server.

To tag images, enter:

Web Monitoring PSA:

docker tag sum-heimdall:<heimdall-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall:<heimdall-tag>
docker tag sum-chrome-agent:<agent-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-chrome-agent:<agent-tag>

API Monitoring PSA:

docker tag sum-heimdall:<heimdall-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall:<heimdall-tag>
docker tag sum-api-monitoring-agent:<agent-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-api-monitoring-agent:<agent-tag>

Replace <REGISTRY_HOST> and <REGISTRY_PORT> to what you configured while deploying the registry.

To push the images, enter:

Web Monitoring PSA:

docker login <REGISTRY_HOST>:<REGISTRY_PORT>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall:<heimdall-tag>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-chrome-agent:<agent-tag>

API Monitoring PSA:

docker login <REGISTRY_HOST>:<REGISTRY_PORT>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall:<heimdall-tag>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-api-monitoring-agent:<agent-tag>

Deploy the Web Monitoring PSA and API Monitoring PSA

Note: Ensure that you follow the applicable sequence of steps when installing Web Monitoring PSA and API Monitoring PSA, respectively; some steps are common for both procedures.

The application is deployed to the cluster after the images are in the Registry. You use the Helm chart to deploy and create all Kubernetes resources in the required order.

Note: Starting from PSA 23.12 release, you must deploy both Ignite and Heimdall in a single namespace named "measurement."

Install Helm following these instructions.
Create a new measurement namespace to run Apache Ignite pods.
Warning: Ensure that you first run the Apache Ignite commands and then run the Heimdall commands.
To create a new measurement namespace, enter:
```
kubectl create namespace measurement
```
Before you deploy Apache Ignite, you must set some configuration options. To view the configuration options, navigate to the previously downloaded ignite-psa.tgz file and enter:
```
helm show values ignite-psa.tgz > values-ignite.yaml
```
Note: If you want to enable persistence, set persistence > enabled. This is an optional configuration
To deploy the Helm chart using the above-mentioned configuration, navigate to the previously downloaded ignite-psa.tgz file and enter:
```
helm install synth ignite-psa.tgz --values values-ignite.yaml --namespace ignite
```
All the Kubernetes resources are created in the cluster, and you can use Apache Ignite. After a few seconds, Apache Ignite initializes and is visible in the Controller.
To verify if the pods are running, enter:
```
kubectl get pods --namespace measurement
```
Proceed to the next steps only after the Apache Ignite pods run successfully.

Using a single command, you can deploy the Helm chart, which contains the deployment details. To deploy the agent, use the Helm chart sum-psa-heimdall.tgz in the zip file that you downloaded previously. Before you deploy the Private Synthetic Agent, you must set some configuration options. To view the configuration options, navigate to the previously downloaded sum-psa-heimdall.tgz file and enter:

helm show values sum-psa-heimdall.tgz > values.yaml

These are the configuration key-value pairs that you need to edit in the values.yaml file:

Web Monitoring PSA Using EC2:


Configuration Key	Value
`heimdall > repository`	`<aws_account_id>.dkr.ecr.<region>. amazonaws.com/sum/sum-heimdall`
`heimdall > tag`	`<heimdall-tag>`
`heimdall > pullPolicy`	`always`
`chromeAgent > repository`	`<aws_account_id>.dkr.ecr.<region>. amazonaws.com/sum/sum-chrome-agent`
`chromeAgent > tag`	`<agent-tag>`
`shepherd > url`	`Shepherd URL`
`shepherd > credentials`	`credentials`
`shepherd > location`	`agent location`

API Monitoring PSA Using EC2:


Configuration Key	Value
`heimdall > repository`	`<aws_account_id>.dkr.ecr.<region>. amazonaws.com/sum/sum-heimdall`
`heimdall > tag`	`<heimdall-tag>`
`heimdall > pullPolicy`	`always`
`apiMonitoringAgent > repository`	`<aws_account_id>.dkr.ecr.<region>. amazonaws.com/sum/sum-api-monitoring-agent`
`apiMonitoringAgent > tag`	`<agent-tag>`
`shepherd > url`	`Shepherd URL`
`shepherd > credentials`	`credentials`
`shepherd > location`	`agent location`

Web Monitoring PSA Using Private Registry:


Configuration Key	Value
`heimdall > repository`	`<REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall`
`heimdall > tag`	`<heimdall-tag>`
`heimdall > pullPolicy`	`always`
`chromeAgent > repository`	`<REGISTRY_HOST>:<REGISTRY_PORT>/sum-chrome-agent`
`chromeAgent > tag`	`<agent-tag>`
`privateRegistry`	`true`
`shepherd > url`	`Shepherd URL`
`shepherd > credentials`	`credentials`
`shepherd > location`	`agent location`

API Monitoring PSA Using Private Registry:


Configuration Key	Value
`heimdall > repository`	`<REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall`
`heimdall > tag`	`<heimdall-tag>`
`heimdall > pullPolicy`	`always`
`apiMonitoringAgent > repository`	`<REGISTRY_HOST>:<REGISTRY_PORT>/sum-api-monitoring-agent`
`apiMonitoringAgent > tag`	`<agent-tag>`
`privateRegistry`	`true`
`shepherd > url`	`Shepherd URL`
`shepherd > credentials`	`credentials`
`shepherd > location`	`agent location`

Note: After configuring using Private Registry

Create registry credentials:

kubectl create secret docker-registry regcred --docker-server=<REGISTRY_HOST>:<REGISTRY_PORT> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email> --namespace measurement

Patch the default service account of the measurement namespace to use the regcred registry credentials:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}' --namespace measurement

You can leave the rest of the values set to their defaults or configure them based on your requirements. See Configure Web Monitoring PSA and API Monitoring PSA for details on shepherd URL, credentials, location, and optional key-value pairs.

Note: You need to replace <aws_account_id> and <region> with your account and region values.

Note:

If the Kubernetes cluster is locked down, and you cannot make cluster-wide configuration, you can make pod-level changes.

For example, if you want to change the pod-level DNS server setting to use your internal nameservers for DNS name resolution, specify the following details in the values.yaml file:


Configuration Key	Value
`agentDNSConfig:`
`enabled:`	`true`
`dnsConfig:`
`nameservers:`	`["4.4.4.4"]`
`searches:`	`["svc.cluster.local", "cluster.local"]`

To deploy the Helm chart using the above-mentioned configuration, navigate to the previously downloaded sum-psa-heimdall.tgz file and enter:
```
helm install heimdall-onprem sum-psa-heimdall.tgz --values values.yaml --namespace measurement
```
All the Kubernetes resources are created in the cluster, and you can use Heimdall. After a few seconds, Heimdall initializes and is visible in the Controller.
To verify if the pods are running, enter:
```
kubectl get pods --namespace measurement
```
To make any changes to the values.yaml after the initial deployment, navigate to the previously downloaded sum-psa-heimdall.tgz file and enter:
```
helm upgrade heimdall-onprem sum-psa-heimdall.tgz --values values.yaml --namespace measurement
```
Warning:
To remove the deployment:

helm uninstall heimdall-onprem --namespace measurement

This is not recommended unless it is required.

Monitor the Kubernetes Cluster

The Helm chart sum-psa-monitoring.tgz in the zip you downloaded installs the monitoring stack. This Helm chart installs kube-prometheus-stack along with a custom Grafana dashboard to monitor the Private Simple Synthetic Agent.

Note: Monitoring the deployment is optional; however, we highly recommend that you monitor the cluster to check its health periodically.

Install the Monitoring Stack

To create a separate monitoringTo review configuration options, enter:
```
kubectl create namespace monitoring
```
To review configuration options, enter:
```
helm show values sum-psa-monitoring.tgz > values-monitoring.yaml
```
This generates a values-monitoring.yaml file that contains all the configuration options. To modify and pass the generated values-monitoring.yaml file while installing the Helm chart, enter:
```
helm install psa-monitoring sum-psa-monitoring.tgz --values values-monitoring.yaml --namespace monitoring
```
After the monitoring stack is installed, you can Launch Grafana (which runs inside the cluster) to view the dashboard. To access Grafana from outside of the cluster, you can configure port forwarding or set up Ingress. To configure port forward to access it locally, enter:
```
kubectl port-forward svc/psa-monitoring-grafana 3000:80 --namespace monitoring
```
Launch localhost:3000 from the browser and log in using the default credentials with username as admin and password as prom-operator. A dashboard named Private Simple Synthetic Agent displays and provides details about the Kubernetes cluster, Apache Ignite, Heimdall, and running measurements.

Upgrade Private Synthetic Agent in Bare Metal K8s

Upgrade the Private Synthetic Agent

To upgrade the PSA, perform the following steps:

Pull the Docker Image

Run the following commands to pull the agent images:

docker pull appdynamics/heimdall-psa
docker pull appdynamics/chrome-agent-psa
docker pull appdynamics/api-monitoring-agent-psa

Add Custom Python Libraries

This is an optional step. In addition to the available standard set of libraries, you can add custom Python libraries to the agent to use in scripted measurements. You build a new image based on the image you loaded as the base image.

Create a Dockerfile and then create RUN directives to run pythonpip. For example, to install the library algorithms you can create a Dockerfile:

# Use the sum-chrome-agent image you just loaded as the base image
FROM appdynamics/chrome-agent-psa:<agent-tag>
USER root
RUN apk add py3-pip
USER appdynamics
# Install algorithm for python3 on top of that
RUN python3 -m pip install algorithms==0.1.4 --break-system-packages

Note: You can create any number of RUN directives to install the required libraries.

To build the new image, run the following commands:
Web Monitoring PSA:
```
docker build -t sum-chrome-agent:<agent-tag> - < Dockerfile
```
API Monitoring PSA:
```
docker build -f Dockerfile-PSA -t sum-api-monitoring-agent:<agent-tag> .
```
You must build the images on the host with the same OS type of Kubernetes cluster nodes. For example, if you are pushing the image to AWS, then run the following command:
```
docker buildx build -f Dockerfile-PSA --platform=linux/amd64 -t sum-api-monitoring-agent:<api-tag> .
```
The newly built agent image contains the required libraries.

Tag and Push Images to the Registry

If you are managing your own Kubernetes cluster, then you must deploy your own registry server. For more details, see Deploy a Registry Server. After deploying the registry, tag and push the images.

Note: Replace <REGISTRY_HOST> and <REGISTRY_PORT> with the values that you used while deploying the registry.

To tag the images, enter:

docker tag sum-heimdall:<heimdall-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall:<heimdall-tag>
docker tag sum-chrome-agent:<agent-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-chrome-agent:<agent-tag>
docker tag sum-api-monitoring-agent:<agent-tag> <REGISTRY_HOST>:<REGISTRY_PORT>/sum-api-monitoring-agent:<agent-tag>

To push the images, enter:

docker login <REGISTRY_HOST>:<REGISTRY_PORT> docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-heimdall:<heimdall-tag>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-chrome-agent:<agent-tag>
docker push <REGISTRY_HOST>:<REGISTRY_PORT>/sum-api-monitoring-agent:<agent-tag>

Update the Helm Chart

Follow these steps and update the configuration key value pairs in the values.yaml file:

Upgrade the PSA

Note: From PSA 23.12 onwards, you must deploy Ignite and Heimdall in a single namespace named measurement.

Navigate to the new Linux distribution folder and run the following command:

helm install synth ignite-psa.tgz --values values-ignite.yaml --namespace measurement

Wait until the status of Ignite pods changes to running. Then, run the following command:

helm upgrade heimdall-onprem sum-psa-heimdall.tgz --values values.yaml --namespace measurement

After the status of the new Heimdall and Ignite pods changes to running, uninstall the old Ignite namespace:
```
helm uninstall synth -n ignite
```