Manage Virtual Appliance Certificates

Audit and renew TLS certificates to maintain connectivity in a Virtual Appliance deployment.

Audit the TLS certificates in your Virtual Appliance cluster regularly so that you can renew certificates before they expire. Certificate renewal helps maintain connectivity among Virtual Appliance services and between the Virtual Appliance and external clients.

Certificate commands

Command Purpose
appdcli platform certificates audit Lists certificate expiration information and severity. In a hybrid deployment, the command also lists supported external certificate files when they are present.
appdcli platform certificates rotate-leaf Renews leaf certificates managed by cert-manager. It does not renew the platform root CA or ingress trust-anchor certificates.
appdcli platform certificates rotate-root-ca Renews the platform root CA. If the deployment uses the default ingress certificate, it also renews that certificate.

Audit Certificates

The audit sorts certificates so that expired certificates and certificates closest to expiration appear first.
Warning: The audit reports the certificate issued by cert-manager or stored in an external certificate file. It does not verify the certificate that a running service currently presents from memory. Most services continue to present an earlier leaf certificate until they are stopped and started.
  1. Run this command to audit certificates.
    CODE 
    appdcli platform certificates audit
    The output includes NAMESPACE, NAME, TYPE, EXPIRES_AT, DAYS_LEFT, and SEVERITY. The type is CA/ANCHOR or LEAF. Virtual Appliance determines the severity based on the days left until the certificate expires.
    Days left Severity
    0 or fewer EXPIRED
    1 through 15 CRITICAL
    16 through 60 WARNING
    More than 60 OK
  2. Optional: For a hybrid deployment, review the customer-managed certificate table, if it appears.

    The command reports these files when they exist in /var/appd/config:

    • hybrid-controller-ca.crt
    • hybrid-mysql-ca.crt
    • hybrid-kafka.crt when hybrid.kafka.defaultCert is false
    Note: Hybrid external certificates are read-only to appdcli. Renew them on the standalone Controller or external systems where they belong to, and then copy the renewed files to the Virtual Appliance.
The command summarizes the cert-manager certificates in each severity and separately lists supported hybrid external certificates.
Sample Output:Sample output of audit certificates

Rotate Leaf Certificates

Renew cert-manager leaf certificates and restart services to load them.

Schedule a maintenance window for the service stop and start.

Use leaf rotation for routine renewal. The command skips the platform root CA and ingress trust-anchor certificates.

  1. Run appdcli platform certificates rotate-leaf.
    The command triggers renewal of all managed leaf certificates, waits for cert-manager to reissue them, and reports any certificate that does not finish reissuing within the wait period.

    Sample Output

    CODE 
    Refreshing 33 leaf certificate(s); 3 trust anchor(s) skipped (root CA + ingress).
  renewal triggered: authn/auth-service (was rev=2)
  renewal triggered: authn/mysql (was rev=2)
  renewal triggered: cisco-controller/controller-certificate (was rev=2)
  renewal triggered: cisco-controller/schema-registry-client-certificate (was rev=2)
  renewal triggered: cisco-dis/cisco-dis-dis-ingestion-svc-tls-secret (was rev=2)
  renewal triggered: cisco-dis/cisco-dis-events-event-ingestion-svc-tls-secret (was rev=2)
  renewal triggered: cisco-dis/cisco-dis-snapshots-snapshot-ingestion-svc-tls-secret (was rev=2)
  renewal triggered: cisco-dis/redis-client-certificate (was rev=2)
  renewal triggered: cisco-eum/eum-certificate (was rev=2)
  renewal triggered: cisco-eum/eum-minio-client-certificate (was rev=2)
  renewal triggered: cisco-eum/eum-redis-client-certificate (was rev=2)
  renewal triggered: cisco-events/events-certificate (was rev=2)
  renewal triggered: cisco-metric/metric-reader-certificate (was rev=2)
  renewal triggered: cisco-metric/metric-rollup-certificate (was rev=2)
  renewal triggered: cisco-monitoring/ma-db-monitor-certificate (was rev=2)
  renewal triggered: cisco-monitoring/ma-pg-client-certificate (was rev=2)
  renewal triggered: cisco-monitoring/ma-redis-client-certificate (was rev=2)
  renewal triggered: cisco-monitoring/monitoring-agent-federate-tls (was rev=2)
  renewal triggered: cisco-monitoring/monitoring-agent-trust-bundle (was rev=2)
  renewal triggered: cisco-synthetic/synthetic-certificate (was rev=2)
  renewal triggered: cisco-synthetic/synthetic-minio-client-certificate (was rev=2)
  renewal triggered: es/es-server-certificate (was rev=1)
  renewal triggered: kafka/kafka-metrics-tls (was rev=1)
  renewal triggered: minio/minio-server-certificate (was rev=1)
  renewal triggered: mysql/mysql-router-certificate (was rev=1)
  renewal triggered: mysql/mysql-server-certificate (was rev=1)
  renewal triggered: postgres/postgres-pgbouncer-client-certificate (was rev=1)
  renewal triggered: postgres/postgres-pgbouncer-server-certificate (was rev=1)
  renewal triggered: postgres/postgres-replication-certificate (was rev=1)
  renewal triggered: postgres/postgres-server-certificate (was rev=1)
  renewal triggered: redis/redis-server-certificate (was rev=1)
  renewal triggered: schema-registry/cp-schema-registry-metrics-tls (was rev=1)
  renewal triggered: schema-registry/schema-registry-server-certificate (was rev=1)
Waiting up to 120s per cert for re-issuance...
  renewed: cisco-dis/cisco-dis-snapshots-snapshot-ingestion-svc-tls-secret rev 2 -> 3
  renewed: cisco-dis/redis-client-certificate rev 2 -> 3
  renewed: cisco-eum/eum-redis-client-certificate rev 2 -> 3
  renewed: cisco-eum/eum-certificate rev 2 -> 3
  renewed: cisco-eum/eum-minio-client-certificate rev 2 -> 3
  renewed: cisco-dis/cisco-dis-events-event-ingestion-svc-tls-secret rev 2 -> 3
  renewed: cisco-events/events-certificate rev 2 -> 3
  renewed: cisco-metric/metric-reader-certificate rev 2 -> 3
  renewed: cisco-monitoring/ma-db-monitor-certificate rev 2 -> 3
  renewed: cisco-monitoring/ma-pg-client-certificate rev 2 -> 3
  renewed: cisco-metric/metric-rollup-certificate rev 2 -> 3
  renewed: cisco-monitoring/ma-redis-client-certificate rev 2 -> 3
  renewed: cisco-monitoring/monitoring-agent-federate-tls rev 2 -> 3
  renewed: cisco-monitoring/monitoring-agent-trust-bundle rev 2 -> 3
  renewed: cisco-synthetic/synthetic-minio-client-certificate rev 2 -> 3
  renewed: es/es-server-certificate rev 1 -> 2
  renewed: kafka/kafka-metrics-tls rev 1 -> 2
  renewed: cisco-synthetic/synthetic-certificate rev 2 -> 3
  renewed: minio/minio-server-certificate rev 1 -> 2
  renewed: mysql/mysql-router-certificate rev 1 -> 2
  renewed: mysql/mysql-server-certificate rev 1 -> 2
  renewed: postgres/postgres-pgbouncer-client-certificate rev 1 -> 2
  renewed: authn/mysql rev 2 -> 3
  renewed: cisco-dis/cisco-dis-dis-ingestion-svc-tls-secret rev 2 -> 3
  renewed: postgres/postgres-pgbouncer-server-certificate rev 1 -> 2
  renewed: postgres/postgres-replication-certificate rev 1 -> 2
  renewed: postgres/postgres-server-certificate rev 1 -> 2
  renewed: redis/redis-server-certificate rev 1 -> 2
  renewed: schema-registry/cp-schema-registry-metrics-tls rev 1 -> 2
  renewed: authn/auth-service rev 2 -> 3
  renewed: cisco-controller/controller-certificate rev 2 -> 3
  renewed: cisco-controller/schema-registry-client-certificate rev 2 -> 3
  renewed: schema-registry/schema-registry-server-certificate rev 1 -> 2

Leaf certificate refresh complete: 33/33 re-issued.

Services still hold the OLD leaf in memory until restarted.
This change requires a stop and start of services. Shall I continue with stop and start? [Y/N]:
  2. At the stop-and-start prompt, enter Y to stop and start all detected, installed subproducts.
    The command detects the installation profile and starts sub products in dependency order. To defer the restart, enter N. During the maintenance window, run appdcli stop <subproduct> && appdcli start <subproduct> <profile> for each affected sub product.
  3. If you accepted the automatic stop and start, run appdcli status application.
  4. Run appdcli platform certificates audit.
    Investigate any certificate renewal is in progress before considering the rotation complete.
After the stop and start, running services load the renewed leaf certificates.

Rotate the Root CA

Renew the platform root CA and, when applicable, the default ingress certificate.

Schedule a maintenance window and prepare to update external client and agent truststores.

CAUTION: External clients and agents that trust the previous root CA or default ingress certificate lose connectivity until you distribute the renewed trust anchors to their truststores.

The command checks ingress.defaultCert in the Virtual Appliance global configuration. A value of true, including the default when the setting has no value, renews the platform root CA and the default self-signed ingress certificate. A value of false renews only the platform root CA. The command does not manage or change the customer-provided ingress certificate.

  1. Run appdcli platform certificates rotate-root-ca.
    CODE 
    WARNING: This will rotate the AppDynamics root CA AND the self-signed ingress certificate.
External clients/agents that pin the previous root CA or ingress certificate in their truststore WILL lose connectivity until the new trust anchor is distributed to every customer environment.
Continue with root CA + ingress certificate refresh? [y/N]: y
  renewal triggered: cert-manager/appd-root-certificate (was rev=1)
  renewal triggered: ingress-master/ingress-certificate (was rev=1)
  renewal triggered: ingress/ingress-nginx-controller-metrics-tls (was rev=1)
Waiting up to 120s per cert for re-issuance...
  renewed: cert-manager/appd-root-certificate rev 1 -> 2
  renewed: ingress-master/ingress-certificate rev 1 -> 2
  renewed: ingress/ingress-nginx-controller-metrics-tls rev 1 -> 2

Trust-anchor refresh complete: 3/3 re-issued.

IMPORTANT — the root CA changed. Existing leaf certificates are still
signed by the OLD CA and will not chain to the new CA.
You should now refresh leaf certificates so they are re-signed by the
new CA (that command will then prompt you to stop/start services):
        appdcli platform certificates rotate-leaf

Also distribute the new root CA / ingress certificate to every
external agent's truststore before they can reconnect.
  2. Review the warning and enter Y to continue.
    Enter N to cancel without renewing certificates.
  3. After the trust-anchor renewal completes, run appdcli platform certificates rotate-leaf so that cert-manager signs all leaf certificates with the new root CA.
  4. At the stop-and-start prompt, enter Y to load the renewed leaf certificates.
    To defer the restart, enter N and stop and start the affected sub products during the same maintenance window.
  5. Distribute the renewed trust material.
    If the deployment uses the default ingress certificate, distribute the new root CA and ingress certificate to every external client and agent truststore that trusted the previous certificates. If the deployment uses a customer-provided ingress certificate, distribute the new platform root CA to the truststores that trust it. The command does not change the customer-provided ingress certificate.
  6. Run appdcli status application.
  7. Run appdcli platform certificates audit.
Running services use the new root CA, renewed leaf certificates, and, when applicable, the renewed default ingress certificate.

For a hybrid deployment, regenerate hybrid-config.tar on the Virtual Appliance, copy it to the Standalone Controller, and run configure.sh on the Standalone Controller. This procedure updates the Kafka CA truststore. If the deployment uses the default ingress certificate, it also updates the schema registry truststore. See Generate the Hybrid Configuration File.