Self Monitoring for the Virtual Appliance

Use Self monitoring to view Virtual Appliance health in the AppDynamics Controller and receive alerts when a monitored component needs attention.

Self monitoring reports Virtual Appliance infrastructure and component metrics to the Controller. It also monitors certificate expiry and maintains Health Rules as the Kubernetes cluster changes. Self monitoring offers the following benefits:
  • View Virtual Appliance health in the same Controller that you use for application monitoring.
  • Receive alerts for node, pod, database, messaging, ingress, and certificate problems.
  • Manage notification actions through one alerting policy.
  • Keep Health Rules aligned with changes to nodes and pods.

Enable Self Monitoring

Configure the Virtual Appliance to report its health metrics and manage Health Rules in the Controller.

Self monitoring starts enabled by default. Configuration for self monitoring lives in the following files under /var/appd/config/:
  • globals.yaml.gotmpl: This file contains a primary switch to turn self-monitoring on or off.
    CODE 
    vaSelfMonitoring:
  enabled: true

  • monitoring.yaml.gotmpl: This file contains Health Rule thresholds, namespace lists, ingress lists, certificate inventories, controller connection settings.

Verify that the Cluster Agent is in the Running state. Check the health of cert-manager and the Virtual Appliance root certificate.

Specify the cluster agent name for self monitoring.
  1. In the Controller, select Servers > Individual Nodes, and copy the Cluster Agent node name.
  2. Open the /var/appd/config/monitoring.yaml.gotmpl file.
  3. Set vaSelfMonitoring.healthRules.clusterAgentName to the copied node name.
    CODE 
    vaSelfMonitoring:
  healthRules:
    clusterAgentName: "appdynamics-cisco-cluster-agent"
Create a policy and attach notification actions.
  1. In the Controller, select Alert & Respond > Policies.
  2. Create the policy named by healthRules.policyName, and attach your notification actions.

    Use va-podmem-policy as the default policy name.

    Note: Newly created Health Rules might take a few minutes to reflect.

    Example:

    CODE 
    healthRules:
  policyName: "va-podmem-policy"
  policySyncEnabled: true
  3. Enter the Controller credentials.
    1. Enter the Controller username and password.

      By default, appdController.monitoringAgentUsername is set to admin and appdController.monitoringAgentPassword to welcome.

      If you change the Controller admin password, change monitoringAgentPassword to match, or the agent stops managing Health Rules. Edit the secrets file to modify these fields. see Edit the secrets.yaml File.

    2. In vaSelfMonitoring.monitoringAccountName, enter the Controller account name.
      If this field is empty, it uses the value entered in appdController.tenantAccountName. See appdController
    Note: Ensure that the account has server monitoring permissions.
  4. Check whether the Cluster Agent is enabled in globals.yaml.gotmpl .
    See enableClusterAgent.
  5. Keep the vaSelfMonitoring block, and set enabled to true.
    CODE 
    vaSelfMonitoring:
  enabled: true

After you apply the Virtual Appliance configuration, Self monitoring reports metrics and creates its Health Rules. Allow a few minutes for the first Controller metric discovery.

Connect Self Monitoring to an external Controller

Send Virtual Appliance Self monitoring metrics and Health Rules to a Controller outside the Kubernetes cluster.

  1. Create the monitoring account on the external Controller.
  2. Open the /var/appd/config/monitoring.yaml.gotmpl file.
  3. Set controllerUrl, monitoringAccountName, and tlsSkipVerify.
    CODE 
    vaSelfMonitoring:
  controllerUrl: "https://controller.example.com:8181"
  monitoringAccountName: "<Controller-account-or-Tenant-Name>"
  4. For a hybrid deployment, set hybrid.controller.monitoringAgentUsername and hybrid.controller.monitoringAgentPassword.
    If you no account name is mentioned in monitoringAccountName, the hybrid deployment uses the account name specified in appdController.tenantAccountName.

Self monitoring uses the external Controller URL and account.

Customize Self Monitoring alerts and notifications

Adjust Health Rule thresholds, monitored resources, and notification actions for your Virtual Appliance.

Self Monitoring manages 14 health rule families. Because four families apply one rule per entity, the Controller typically runs 40 to 60 rules, depending on the pod count. All rules are critical-only (no warning conditions) and use the va-hr- prefix.

Defaults for evaluation windows come from healthRules.defaults:
CODE 
defaults:
  useDataFromLastNMinutes: 5     # evaluation window
  waitTimeAfterViolation: 5      # minutes before re-alerting
Note: You can change the defaults for evaluation window and re-alerting if required.

Self Monitoring manages 14 Health Rule families. All rules create critical violations only. By default, each rule evaluates the last five minutes of data and waits five minutes before sending another alert.

Rule Creates a violation when Default threshold Scope
clusterAgent Cluster Agent availability drops. < 1 Cluster Agent node
infraviz Machine Agent availability drops. < 1 All nodes
nodeCpu Node CPU usage is high. > 80% All nodes
nodeMemory Node memory usage is high. > 80% All nodes
nodeDisk Disk volume usage is high. > 80% Each configured volume on every node
pgReplLag PostgreSQL replication lag is high. > 100 MB Monitoring node
redisMemory Total Redis memory usage is high. > 1500 MB Monitoring node
ingress5xx An ingress reports too many HTTP 5xx errors. > 10 per minute Each configured ingress
kafkaUnderRepl Kafka has an under-replicated partition. > 0 Monitoring node
mysqlMembers A main MySQL cluster member is not ONLINE. > 0 Monitoring node
mysqlAuthMembers An authentication MySQL cluster member is not ONLINE. > 0 Monitoring node
schemaRegistry Schema Registry reports too many errors. > 5 per minute Monitoring node
podMemory A monitored pod stops reporting memory. < 1 MB Each eligible pod
certExpiry A monitored TLS certificate approaches expiry. < 30 days Each configured certificate

Edit Health Rules

You can customize Health Rule names, thresholds, and other parameters in monitoring.yaml.gotmpl. Self Monitoring uses the following naming pattern:

CODE 
<prefix><rule-identifier>[-<entity>]

The default prefix is va-hr-. You can change the prefix or use hrIdentifier to change a rule identifier. The prefix can contain no more than 30 characters.

Rules that monitor the cluster or a single node do not include an entity suffix. Fan-out rules include the name of the monitored entity.

For example, you can increase the nodeCpu threshold from 80 percent to 90 percent and specify a custom rule identifier:

CODE 
vaSelfMonitoring:
  healthRules:
    rules:
      nodeCpu:
        enabled: true
        hrIdentifier: "<custom-rule-identifier>"
        compareValue: 90

With this configuration, the rule creates a violation when node CPU usage exceeds 90 percent.

Policy behavior

You can create a policy that includes one or more Health Rules. The agent manages the policy membership. During reconciliation, it removes any Health Rules that you added manually.

Every reconciliation replaces the named policy's Health Rule list with enabled, prefix-matching self-monitoring rules. Unrelated rules added to that policy are removed. Reconciliation runs every 300 seconds by default.

Note: You can change the reconciliation interval in healthRules.reconcileInterval. Enter the interval in seconds.

Monitor Java-Based Applications

The Virtual Appliance includes Java services such as authn, authz, cisco-eum, cisco-events, cisco-aiops, cisco-uil, cisco-synthetic, and cisco-metric.

To monitor these services with Java Agent, set javaInstrumentationOfServices to true. This setting injects the Java Agent into the Java services.

Note: Java application monitoring consumes Java APM licenses. Ensure that sufficient valid licenses are available and applied before you turn on this feature.
  1. Open the /var/appd/config/monitoring.yaml.gotmpl file.
  2. Change a rule threshold under healthRules.rules.<name>.
    CODE 
    vaSelfMonitoring:
  healthRules:
    rules:
      nodeCpu:
        enabled: true
        compareValue: 90
    Table 1. Health Rule configuration fields
    Field Description
    enabled false deletes that rule's Health Rules from the Controller.
    compareCondition GREATER_THAN_SPECIFIC_VALUE or LESS_THAN_SPECIFIC_VALUE.
    compareValue Threshold. MB for pgReplLag and redisMemory; literal units elsewhere.
    evaluateToTrueOnNoData Whether missing data is itself a violation. Required.
    hrIdentifier Overrides the Health Rule name component.
  3. Set a rule's enabled value to false when you want Self monitoring to delete that rule's Health Rules during the next reconciliation.
  4. Replace a namespaces, ingresses, certificates, or volumes list only when you want to replace the complete default list.
    Note: Self Monitoring does not monitor every certificate. By default, it monitors one configured certificate per namespace. Review and customize the certificates list to cover additional certificates.

    Do not use an empty list to turn off a rule. Set the rule's enabled value to false.

  5. Set redisMemory.compareValue to a useful fraction of total Redis capacity.

    The default memory threshold is set to 1500 MB across the cluster nodes.

    Table 2. Sizing Guidance
    Profile Max Memory Per Node Cluster Capacity (3 Nodes)
    Small 512 MB ~1.5 GB
    Medium 384 MB ~1.15 GB
    Large 2.5 GB ~7.5 GB
    Extra Large 5 GB ~15 GB

    Use about 80 percent as a starting point. Adjust the value if you change the Redis replica count.

  6. In the Controller, select Alert & Respond > Policies, create the policy named by healthRules.policyName, and attach notification actions.
  7. Make future threshold and scope changes in monitoring.yaml.gotmpl.

    Do not edit managed Health Rules in the Controller. Self monitoring replaces those edits during reconciliation.

To stop health rule reconciliation, set healthRules.enabled to false. Existing health rules remain in the Controller and continue to evaluate until you remove them manually or turn reconciliation back on.

During an upgrade or restore, your settings in monitoring.yaml.gotmpl override the new release defaults, so your customization remain. Custom lists replace the default lists and do not receive new entries added by the release, so review them after each upgrade. Keep vaSelfMonitoring as a valid mapping because commenting it out or replacing it with another value causes the restore to fail.

Turn off Self Monitoring and Health Rules

Stop the Virtual Appliance from collecting Self monitoring metrics and managing its Health Rules.

Stop the Virtual Appliance from collecting Self monitoring metrics and managing its Health Rules.

Turn off Health Rules

Use the appropriate setting based on whether you want to remove individual Health Rules or stop Self Monitoring management.

Setting Result
rules.<name>.enabled: false Deletes the Health Rules for that rule during the next reconciliation.
healthRules.enabled: false Stops Health Rule reconciliation. Existing Health Rules remain in the Controller and continue to evaluate.
vaSelfMonitoring.enabled: false Turns off Self Monitoring and prevents the monitoring agent from running. Existing Health Rules remain in the Controller.
policySyncEnabled: false Stops the agent from managing policy membership. It does not turn off or delete Health Rules.
Note: Make Health Rule changes in monitoring.yaml.gotmpl. The agent replaces manual Controller changes during reconciliation.
Turn off Self Monitoring
  1. Open the /var/appd/config/globals.yaml.gotmpl file.
  2. Set vaSelfMonitoring.enabled to false.
    CODE 
    vaSelfMonitoring:
  enabled: false

The Virtual Appliance stops the Self monitoring collection and alerting pipeline. Existing Health Rules remain in the Controller.