Controller SSL and Certificates

The Controller comes with a preconfigured HTTPS port (port 8181 by default) that is secured by a self-signed certificate. This page describes how to replace the default certificate with your own custom certificate.

About Controller SSL and Certificates

For production use, we recommend that you replace the self-signed certificate with a certificate signed by a third-party Certificate Authority (CA) or your own internal CA. If you are deploying .NET Agents, you must replace the self-signed certificate with one signed by a CA, since the .NET agents do not work with self-signed certificates.

Controller SSL Certificates

You can manage your Controller SSL certificate using the command line or the Enterprise Console UI under Configurations. On the UI, the Appserver Configurations and Reports Service Configurations and Fleet Management Configurations pages both contain sections that display the SSL certificate information and provide an Edit Certificateo ption.

See Import an Existing Keypair into the Keystore Using the Command Line and Import an Existing Keypair into the Keystore Using the Enterprise Console UI.

Controller Keystore and Artifacts

This page describes how to replace the existing key in the default keystore. Replacing the entire keystore is not recommended unless you first export the existing artifacts from the default keystore and import them into your own keystore.

The default Controller keystore includes the following artifacts:

  • s1as: A self-signed private key provided with the Glassfish application server used by the Controller for secure communication on port 8181.
  • reporting-instance: A private key used by the reporting service, the service that enables scheduled reports.

Update Keystore Passwords

You can modify the password for the keystore.jks and cacert.jks files that are used to generate the keystore artifacts. The password for both files must be the same.

You cannot modify, however, the password for the reporting-service.pfx file that is generated by the keystore artifact reporting-instance and used by the Reporting Service.

How to View the Keystore

You can view the contents of the keystore yourself using the keytool utility in your jre/<version>bin directory. Enter the default keystore password changeit when prompted.

keytool -list -v -keystore controller_home/appserver/jetty/etc/ keystore.jks

The exact steps to implement security typically vary depending on the security policies for the organization. For example, if your organization already has a certificate to use, such as a wildcard certificate used for your organization's domain, you can import the existing certificate into the Controller keystore. Otherwise, you'll need to generate a new one along with a certificate signing request. The following sections take you through these scenarios.

Before Starting

The following instructions describe how to configure SSL using the Java keytool utility bundled with the Controller installation. You can find the keytool utility in a directory at the same level as <controller_home>. In other words, from <controller_home>, navigate to ../jre/<version>/bin:

The steps assume that the keytool is in the operating system's path variable. To run the commands as shown, you first need to put the keytool utility in your system's path. Use the method appropriate for your operating system to add the keytool to your path.

While the directory paths in this topic use forward slashes, the instructions apply to both Linux and Windows Operating System environments. The steps note where there are differences in the use of commands between operating systems.

Create a Certificate and Generate a CSR

If you don't have a certificate to use for the Controller, create it as follows.

Splunk AppDynamics On-Premises requires using a X.509 digital certificate, which works with any file type.

In these steps, you generate a new certificate within the Controller's active keystore, so it has immediate effect.

The steps are intended to be used in a staging environment, and require the Controller to be shut down and restarted. Alternatively, you can generate the key as described here but in a temporary keystore rather than the Controller's active keystore. After the certificate is signed, you can import the key from the temporary keystore to the Controller's keystore.

  1. At a command prompt, change directories to the following location:
    <Controller_home>/appserver/jetty/etc
  2. Create a backup of the keystore file. For example, on Linux, you can run:On Windows, you can use the copy command in a similar manner.
    cp keystore.jks keystore.jks.backup
  3. If it's still running, stop the Controller.
  4. Delete the existing certificate with the alias s1as from the keystore:
    keytool -delete -alias s1as -keystore keystore.jks
  5. Create a new key pair in the keystore using the following command. This command creates a key pair with a validity of 1825 days (5 years). Replace 1825 with the validity period appropriate for your environment, if desired.
    keytool -genkeypair -alias s1as -keyalg RSA -keystore keystore.jks -keysize 2048 -validity 1825
    Follow the on-screen instructions to configure the certificate. Note that:
    • For the first and last name, enter the domain name where the Controller is running, for example, controller.example.com.
    • Enter the default password for the key, changeit .

    This generates a self-signed certificate in the keystore. We'll generate a signing request for the certificate next. You can now restart the Controller and continue to use it. Since it still has a temporary self-signed certificate, browsers attempting to connect to the Controller UI will get a warning to the effect that its certificate could not be verified.

    See Change Keystore Password for information on changing the default password for the keystore and certificates.

  6. Generate a certificate signing request for the certificate you created as follows:
    keytool -certreq -alias s1as -keystore keystore.jks -file AppDynamics.csr
  7. Submit the certificate signing request file generated by the command (AppDynamics.csr in our example command) to your Certificate Authority of choice.When it's ready, the CA will return the signed certificate and any root and intermediary certificates required for the trust chain. The response from the Certificate Authority should include any special instructions for importing the certificate if needed. If the CA supplies the certificate in text format, just copy and paste the text into a text file.
  8. Import the signed certificate:This command assumes the certificate is located in a file named mycert.cer.
    keytool -import -trustcacerts -alias s1as -file mycert.cer -keystore keystore.jks
  9. If you get the error "Failed to establish chain from reply", install the issuing Certificate Authority's root and any intermediate certificates into the keystore. The root CA chain establishes the validity of the CA signature on your certificate. Although most common root CA chains are included in the bundled JVM's trust store, you may need to import additional root certificates, such as certificates belonging to a private CA. To do so:
    keytool -import -alias [Any_alias] -file <path_to_root_or_intermediate_cert> -keystore <Controller_home>/appserver/jetty/etc/cacerts.jks
When done importing the certificate chain, try importing the signed certificate again.

Import an Existing Keypair into the Keystore Using the Command Line

These steps describe how to import an existing public and private key into the Controller keystore. We'll step through this scenario assuming that the existing public and private keys need to be converted to a format compatible with Java Keystore, say from DER format to PKCS#12. You'll need to use OpenSSL to combine the public and private keys, and then use keytool to import the combined keys into the Controller's keystore.

Most Linux distributions include OpenSSL. If you are using Windows or your Linux distribution does not include OpenSSL, you may find more information on the OpenSSL website.

This assumes that we have the following files:

  • private key: private.key
  • signed public key: cert.crt
  • CA root chain: ca.crt

The private key you use for the following steps must be in plain text format. You must assign the default password (changeit) to the private key when you convert it to PKCS12 keystore format.

  1. Use OpenSSL to combine your existing private key and public key into a compatible Java keystore:

    openssl pkcs12 -inkey private .key -in cert.crt -export -out keystore.p12
  2. If the Controller is still running, stop it.

  3. Change to the keystore directory:

    cd <Controller_home>/appserver/jetty/etc/

  4. Create a backup of the keystore file. For example, on Linux, you can run:

    cp keystore.jks keystore.jks.backup

    On Windows, you can use the copy command in a similar manner.

  5. Delete the self-signed certificate with alias s1as from the default keystore:

    keytool -delete -alias s1as -keystore keystore.jks

  6. Import the PKCS #12 key into the default keystore:

    keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype JKS

  7. Update the alias name on the key pair you just imported:

    The alias name should be s1as. Do not change it from this name.

    keytool -changealias -alias "1" -destalias "s1as" -keystore keystore.jks

  8. Change the password of the imported private key:

    keytool -keypasswd -keystore keystore.jks -alias s1as -keypass <.p12_file_password> -new <password>

    For the new private key password, use the default (changeit) or the master password set as described in Change Keystore Password, if changed.

  9. If you get the error "Failed to establish chain from reply", install the issuing Certificate Authority's root and any intermediate certificates into the keystore. The root CA chain establishes the validity of the CA signature on your certificate. Although most common root CA chains are included in the cacerts.jks truststore, you may need to import additional root certificates. To do so:

    keytool -import -alias <Any_alias> -file <path_to_root_or_intermediate_cert> -keystore <Controller_home>/appserver/jetty/etc/cacerts.jks

    When done, try importing the signed certificate again.

  10. Start the Controller.

Import an Existing Keypair into the Keystore Using the Enterprise Console UI

  1. Log in to the Enterprise Console.
  2. Navigate to Configurations > Controller Settings > Appserver Configurations.
  3. Click the SSL Certificate Management tab.
  4. Click Edit Certificate.
  5. In the Edit Certificate pop-up window, add the Certificate Body, Certificate Chain (optional), and Private Key.
  6. In the Edit Certificate pop-up window, click Save.
  7. On the Appserver Configurations age, click Save.
  8. When the Appserver Configurationspop-up window appears, click OK.
  9. Wait until the Controller Configurations Update job finishes. You can review its status on the Jobs page.

Verify the Use of SSL

To make sure the configuration works, use a browser to connect to the Controller over the default secure port, port 8181:

https: //<controller_host>:8181/controller

Make sure the Controller entry page loads in the browser correctly. Also, verify that the browser indicates a secure connection. Most browsers display a lock icon next to the URL to indicate a secure connection.

After changing the certificate on the Controller, you will need to import the public key of the certificate to the agent truststore. For information on how to do this, see the topic specific for the agent type:

If there is no proxy configured and the agent is reporting to the Controller itself, then the following changes are also mandatory:

  1. Run the following command:
    platform-admin.sh stop-controller-appserver
    On Windows, run this command from an elevated command prompt (which you can open by right-clicking on the Command Prompt icon in the Windows Start menu and choosing Run as administrator): 
    platform-admin.exe cli stop-controller-appserver
  2. Search for the following properties in <controller_home>/appserver/jetty/start.d/start.ini, and replace the port with the SSL port, as the non-secure port is disabled:
    -Dappdynamics.controller.port=
-Dappdynamics.controller.services.port=
  3. In the following property, change the protocol from HTTP to HTTPS, and change the port to the secure port.You can also use REST API to update the deeplink URL: 
    - Dappdynamics.controller.ui.deeplink.url=
    curl -k --basic --user root@system --header "Content-Type: application/json" --data '
{ "controllerURL": "https://<controller>:<ssl_port>" }' https://<controller>:<ssl_port>/controller/rest/accounts/<ACCOUNT-NAME>/update-controller-url
  4. Add the following JVM argument anywhere above or below the above JVM arguments to ensure the internal agent connects using SSL. 
    -Dappdynamics.controller.ssl.enabled=true
  5. Run the following command:
    platform-admin.sh start-controller-appserver

    On Windows, run the following in an elevated command prompt:

    platform-admin.exe cli start-controller-appserver
You can also use the modifyJVMOptions.sh script to make the changes.

Change Keystore Password

The default password for the keystore used by the Controller is changeit. This is the default password for the Jetty keystore, and is a well-known (and thus insecure) password. For a secure installation, you need to change it.

Changing keystore password should include setting the same passwords for all the keys as well.

By default, keystore.jks contains s1as and reporting-instance keys.

  1. Update the keystore password: 
    <JRE_HOME>/bin/keytool -storepasswd -keystore <controller_home>/appserver/jetty/etc/keystore.jks -storepass <current_password> -new <new_password>
  2. Update the truststore password: 
    <JRE_HOME>/bin/keytool -storepasswd -keystore <controller_home>/appserver/jetty/etc/cacerts.jks -storepass <current_password> -new <new_password>
  3. Update the password for keys: 
    <JRE_HOME>/bin/keytool -keypasswd -keystore <controller_home>/appserver/jetty/etc/keystore.jks -storepass <new_password> -alias s1as -keypass <current_password> -new <new_password>
<JRE_HOME>/bin/keytool -keypasswd -keystore <controller_home>/appserver/jetty/etc/keystore.jks -storepass <new_password> -alias reporting-instance -keypass <current_password> -new <new_password>
  4. Create obfuscated password for the keystore password <new_password>:
    <JRE_HOME>/bin/java -jar <controller_home>/tools/lib/scs-tool.jar obfuscate -plaintext <new_password>
    This command creates the obfuscated password. For example:
    Example obfuscated password: s_-001-12-H8v0OuZ2X/M=SOMM06ufKVOATetbV2BYxQ==
  5. Update the obfuscated password in the Enterprise Console UI:
    1. Navigate to Configurations > Controller Settings > Appserver Configurations.
    2. In the JVM Options tab, update the following sections under SSL Context Config:
      <Set name="KeyStorePassword">
<Call class="com.singularity.ee.util.security.credentialstore.ObfuscationWrapper" name="deobfuscateString">
<Arg>[Obfuscated Password]</Arg>
</Call>
</Set>
<Set name="TrustStorePassword">
<Call class="com.singularity.ee.util.security.credentialstore.ObfuscationWrapper" name="deobfuscateString">
<Arg>[Obfuscated Password]</Arg>
</Call>
</Set>
<Call class="java.lang.System" name="setProperty">
<Arg>javax.net.ssl.keyStorePassword</Arg>
<Arg>
<Call class="com.singularity.ee.util.security.credentialstore.ObfuscationWrapper" name="deobfuscateString">
<Arg>[Obfuscated Password]</Arg>
</Call>
</Arg>
</Call>
<Call class="java.lang.System" name="setProperty">
<Arg>javax.net.ssl.trustStorePassword</Arg>
<Arg>
<Call class="com.singularity.ee.util.security.credentialstore.ObfuscationWrapper" name="deobfuscateString">
<Arg>[Obfuscated Password]</Arg>
</Call>
</Arg>
</Call>
    3. Click Save.
  6. Update the new keystore password in Enterprise Console:
    1. Navigate to Configurations > Controller Settings > Appserver Configurations.
    2. In SSL Certificate Management, update the new Controller Keystore Password and confirm.
    3. Click Save.

Updating an Expired Certificate

The steps to renew an expired or soon-to-expire certificate are similar to those for replacing the default certificate, as documented in Create a Certificate and Generate a CSR. To update the expired certificate:

  1. Back up the existing keystore.
  2. At a command prompt, change directories to the following location:
    <controller_home>/appserver/jetty/etc/
  3. Create a backup of the keystore file.
    1. On Linux, you can run the following command:
      cp keystore.jks keystore.jks.backup
    2. On Windows, you can use the copy command in a similar manner. If the controller is still running, stop the controller.
  4. Since you already have a Java keystore, run the following command to issue a certificate signing request. You should use this keystore for the csr not create a new one. You will be importing the new certificate into this keystore.
    keytool -certreq -alias s1as -keystore keystore.jks -file AppDynamics.csr
  5. Submit the certificate signing request file Appdynamics.csr generated by the above example command to your Certificate Authority of choice.
    1. When it's ready, the Certificate Authority will return the signed certificate and any root and intermediary certificates required for the trust chain.
    2. The response from the Certificate Authority should include instructions for importing the certificate if needed.

      If the Certificate Authority supplies the certificate in text format, copy and paste the text into a text file.

  6. You can list out the obtained certificate as follows if it is not in text format.
    keytool -printcert -v -file <your obtained certificate>
  7. Import the signed certificate obtained into the keystore that you already have.
    keytool -import -alias s1as -file <your obtained certificate> -keystore keystore.jks
    1. The imported certificate will replace the old one, provided you use the same alias as the previous one.
    2. Sometimes the root and intermediate certificates of the certification authority are also expired. If that's the case, you will see the message Failed to establish chain from reply.
  8. If the root and intermediate certificates of the certification authority are expired, they also have to be imported in your cacerts.jks so that the chain of trust can be established. You can follow your certification authority's instructions to download the root and intermediate certificates.
    1. Keep the same alias as before for root and intermediate when you import these certificates into cacerts.jks
      keytool -import -alias <previous alias used for the certificate> -file <path_to_root_or_intermediate_cert> -keystore <Controller_home>/appserver/jetty/etc/cacerts.jks

Trust Stores and Keystores

  • Java trust store, cacerts, contain root certificates of well-known certification authorities. The validity of a certificate presented during the TLS/SSL (Transport Layer Security/Secure Sockets Layer) session is checked from cacerts.jks . There are no private keys or passwords in cacerts. They will contain the intermediate and root certificates of certification authorities.
  • Java Keystore is used to store private key and the identify certificate for the server, which means that the keystore is used to store your server’s credentials.

Troubleshoot Certificate Issues

You may encounter issues when you use default self-signed certificates. Follow these instructions to troubleshoot the issues.

Issue: 'net::ERR_CERT_COMMON_NAME_INVALID'

net::ERR_CERT_COMMON_NAME_INVALID '

This error is displayed even though you import the the default self-signed certificate into "Certificates" of [Trusted Root Certification Authorities] of a Windows machine. Because, the Chrome 58 or higher and Edge browsers do not trust the TLS certificates if the Subject Alternative Name (SAN) is absent.

To troubleshoot this issue, you must regenerate the certificate with SAN and import into "Certificates" of [Trusted Root Certification Authorities] of a Windows machine:

  1. Back up the following keystore.jks file.
    <AppD Home>\Platform\product\controller\appserver\jetty\etc\keystore.jks
  2. Delete the s1as entry from keystore.jks .
    keytool -delete -alias s1as -keystore <AppD Home>\Platform\product\controller\appserver\jetty\etc\keystore.jks
  3. Create a new key-pair and the s1as entry in keystore.jks , and add it to keystore.jks .
    keytool -genkeypair -alias s1as -keyalg RSA -keystore <AppD Home>\Platform\product\controller\appserver\jetty\etc\keystore.jks -keysize 2048 -validity 1825 -storepass changeit -keypass changeit -dname "CN=<hostname>" -ext "SAN=dns:<hostname>"
  4. Export the certificate from keystore.jks .
    keytool -exportcert -alias s1as -keystore keystore.jks -file <AppD Home>\Platform\product\controller\appserver\jetty\etc\keystore_controller.crt -rfc -storepass changeit

  5. Open the keystore_controller.crt Certificate Manager, select Certificates under Trusted Root Certification Authorities, and import

  6. Restart the Controller.