Encrypt Events Service Credentials

Configure the Secure Credential Store to encrypt the Events Service keystore password.

You can use the Secure Credential Store to avoid storing supported Events Service passwords in plain text in the events-service-api-store.properties file.

This procedure shows how to encrypt the ad.es.client.keystore.password property.

If you are configuring basic authentication and TLS, first see Set Up TLS To Encrypt Communications in the Elastic Stack for instructions to create the Events Service keystore and set the ad.es.client.keystore.password property. Then, use this procedure to replace the plain-text password with an encrypted value.

Note: This procedure applies to non-FIPS Splunk AppDynamics On-Premises Events Service deployments, including releases earlier than 26.7.0. For a FIPS-enabled deployment, use the FIPS-supported secure credential store configuration.

Before you begin

Before you configure the Secure Credential Store:

  • Schedule a maintenance window.

  • Back up events-service-api-store.properties.

  • If a secure credential keystore already exists, back it up and do not replace it.

  • Locate scs-tool.jar in the Controller installation:

    CODE 
    <controller_home>/tools/lib/scs-tool.jar
    The scs-tool.jar utility is installed with the Controller. You can run the utility on the Controller host or copy it to a secure administrative host. The resulting secure credential keystore must be available on every Events Service node that uses the encrypted value.

  • Locate a supported Java runtime.

  • Ensure that the account running the Events Service can read the secure credential keystore.

  • Store the credential-store password in an approved password manager.

Note: The -storepass and -plaintext arguments can expose sensitive values in command history or process information. Run the commands in a controlled environment and follow your organization’s requirements for handling command-line secrets.
Define the file locations
  1. Set variables for the utility, Java runtime, secure credential keystore, and Events Service properties file. Adjust the paths for your installation.
    CODE 
    SCS_TOOL=<controller_home>/tools/lib/scs-tool.jar
JAVA=<java_home>/bin/java
SCS_KEYSTORE=<events_service_home>/processor/conf/security/scs.keystore
PROPERTIES=<events_service_home>/processor/conf/events-service-api-store.properties
  2. Verify that Java and the Secure Credential Store utility are available.
    CODE 
    # Confirm Java is available
"$JAVA" -version
# Confirm Secure Credential Store utility works
"$JAVA" -jar "$SCS_TOOL" -h
    The utility help begins with:
    CODE 
    Here are possible command line options:
Create or validate the secure credential keystore
  1. If the secure credential keystore does not exist, create its parent directory.
    CODE 
    mkdir -p <events_service_home>/processor/conf/security
  2. Create the keystore.
    CODE 
    "$JAVA" -jar "$SCS_TOOL" generate_ks \
  -filename "$SCS_KEYSTORE" \
  -storepass <credential-store-password>
  3. If the keystore already exists, validate it.
    CODE 
    "$JAVA" -jar "$SCS_TOOL" validate_ks \
  -filename "$SCS_KEYSTORE" \
  -storepass <credential-store-password>
    Important: Keep both the keystore and its password. Encrypted values cannot be recovered if the keystore or its password is lost.
  4. Restrict access to the keystore to the operating-system account that runs the Events Service.
  5. For a multi-node Events Service cluster, securely copy the same keystore to every node that uses the encrypted property.
    Store the keystore at the path configured by ad.secure.credential.store.filename on each node.
Encrypt the Events Service keystore password
  1. Run the following command. Replace <events-service-keystore-password> with the current value of ad.es.client.keystore.password.
    CODE 
    $JAVA -jar $SCS_TOOL encrypt -filename $SCS_KEYSTORE -storepass <storepass> -plaintext <plaintext-password>
    The command returns an encrypted value similar to:
    CODE 
    -001-24-uwxSfkpNa/c+...
  2. Add the secure:// prefix when you place the encrypted value in the Events Service properties file.
    CODE 
    ad.es.client.keystore.password=secure://<encrypted-value>
    CAUTION: Do not add the secure:// prefix to the value passed to the encrypt command.
Obfuscate the credential-store password
  1. Obfuscate the password that protects the secure credential keystore.
    CODE 
    "$JAVA" -jar "$SCS_TOOL" obfuscate \
  -plaintext <credential-store-password>
    The command returns an obfuscated value similar to:
    CODE 
    s_-001-12-n8+5fe5xrjU=vZjkDk6R8It+...
    Use the complete output as the value of ad.secure.credential.store.password.
Configure the Events Service
  1. Open the Events Service properties file for editing.
    <events_service_home>/processor/conf/events-service-api-store.properties
  2. Add the secure credential store location and its obfuscated password.
    CODE 
    # Path to the SCS keystore
ad.secure.credential.store.filename=/opt/appdynamics/platform/events-service/processor/conf/security/scs.keystore

# Obfuscated keystore password (full output of Step 4, e.g. s_-001-12-...)
ad.secure.credential.store.password=<full-output-of-obfuscate-command>
  3. Replace the plain-text Events Service keystore password.
    CODE 
    # Before
ad.es.client.keystore.password=myPlainTextPassword

# After — prepend secure:// to the output of the encrypt command
ad.es.client.keystore.password=secure://<encrypted-value-from-step-8>
    The resulting configuration has the following form:
    CODE 
    ad.secure.credential.store.filename=<events_service_home>/processor/conf/security/scs.keystore
ad.secure.credential.store.password=s_-001-12-...
ad.es.client.keystore.password=secure://-001-24-...
  4. Verify the secure credential store configuration.
    Note: ad.secure.credential.store.password uses the complete output of the obfuscate command. Do not add secure:// to this property. Only supported encrypted properties, such as ad.es.client.keystore.password, use the secure:// prefix.
Restart and verify the Events Service
  1. You must restart the Events Service to apply the changes. Restart the Events Service by using the Enterprise Console or the applicable command-line procedure.
    See Start and Stop the Events Service.
  2. After the restart, verify the Events Service configuration.
    1. Verify that every Events Service node is healthy.
    2. Confirm that the Controller can connect to the Events Service.
    3. Check the Events Service log for secure credential store, keystore, decryption, authentication, or configuration errors.
    4. Confirm that the Events Service accepts requests over the configured secure connection.
The Events Service keystore password is encrypted in the Events Service properties file.
If the Events Service does not start, restore the backed-up properties file and secure credential keystore, and then restart the service. Do not generate a replacement keystore unless you also re-encrypt every value that depends on the original keystore.