Set Up TLS To Encrypt Communications in the Elastic Stack

To secure the communication between nodes in a cluster, you encrypt it using Transport Layer Security (TLS). This basic security setup helps prevent unauthorized nodes from accessing your cluster.

In a secured cluster, the Elastic search nodes must use a certificate signed by certificate authority (CA) to identify themselves before initiating a communication with other nodes. The cluster must validate the authenticity of these certificates. A certificate authority (CA) is a trusted entity that issues digital certificates. These certificates are data files used to cryptographically link an entity with a public key.

Certificates are not bundled with the Events Service. You can use use your own CA to enable security on an existing, unsecured cluster, or manually configure TLS on the transport interface. You can also secure the HTTP traffic using TLS.

Events Service version 23.4.0 supports the following software versions and variables:

AuthenticationBasic Auth
TLS versionTLSv1.3
APPLICATION_HOME (variable)/home/ec2-user/appdynamics/platform/product/events-service/processor

Enable Basic Authentication and TLS for Transport Interface

If your cluster has multiple nodes, you must enable basic authentication and then configure Transport Layer Security (TLS) between nodes. The basic authentication is not sufficient for production mode clusters.

Note: Authentication is not supported for ElasticSearch internode communication.
  1. Stop the Events Service server.
  2. Enable basic authentication on every node in the cluster as follows:

    On every node

    A. export ES_JAVA_HOME=$APPLICATION_HOME/jre
B. cd $APPLICATION_HOME/elasticsearch
C. chmod +x bin/elasticsearch-keystore
D. bin/elasticsearch-keystore add bootstrap.password # Enter the password for 'elastic' superuser. Save this as this will be needed for any operation on elsticsearch and will be used by events-service to authenticated the requests to elasticsearch
E. cd $APPLICATION_HOME/conf/security # mkdir if doesn't exist
F. keytool -importpass -alias elastic_user_password -keystore events-service.keystore -storetype pkcs12 # create a keystore and store the elastissearch 'elastic' user password from Step D.  Also, Enter keystore password that will be set in ad.es.client.keystore.password property

G. vi $APPLICATION_HOME/conf/events-service-api-store.properties
   -ad.es.node.security.basic.auth.enabled=true # Enables basic Auth 
   -ad.es.client.basic.auth.enabled=true #Enables Java client to connect to the ElasticSearch cluster that has basic auth enabled  
   -ad.es.client.keystore.password= # Set the password from Step F
  3. Enable basic authentication for a single node setup (typically used for testing) as follows:

    For single node setup (for testing)

    A. vi $APPLICATION_HOME/conf/events-service-api-store.yml 
B. Add 'discovery.type: single-node' property in nodeSettings section  
 - className: com.appdynamics.analytics.processor.elasticsearch.configuration.ElasticsearchConfigManagerModule
     properties:
        nodeSettings:
C. vi $APPLICATION_HOME/conf/events-service-api-store.properties
      -ad.es.cluster.initial_master_nodes=  # set this as Empty 
      -ad.es.node.unicast.hosts= # set this as Empty
  4. Enable TLS on transport interface.
    Note:

    The property ad.es.node.security.tls.transport.enabled enables or disables TLS for internode communication.

    Ensure that:

    before you generate the certificate.

    Generate CA certificate and node certificate as follows:

    Generate certificate

    On any one node, Generate CA certificate by following step A to D
A. export ES_JAVA_HOME=$APPLICATION_HOME/jre
B. cd $APPLICATION_HOME/elasticsearch
C. chmod +x bin/elasticsearch-certutil
D. bin/elasticsearch-certutil ca  #Enter pwd: test123,  Should see file name elastic-stack-ca.p12
E. Copy the same elastic-stack-ca.p12 CA cert to $APPLICATION_HOME/elasticsearch dir on every node in cluster.
F. On every node in cluster, generate node certificate
   - cd $APPLICATION_HOME/elasticsearch
   - bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --dns {node_hostname} #should see file elastic-certificates.p12 in current dir, node_hostname can be found by running hostname cmd
   - cp elastic-certificates.p12 config/cert
  5. Enable the TLS on every node by editing the properties as follows:

    On every node, Enable TLS

    a. export ES_JAVA_HOME=$APPLICATION_HOME/jre
b. vi $APPLICATION_HOME/conf/events-service-api-store.properties
   - ad.es.node.security.tls.transport.enabled=true #Set this property to enable TLS for transport
c. cd $APPLICATION_HOME/elasticsearch
d. chmod +x bin/elasticsearch-keystore
e. bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password #Enter node certificate password from step 4.F
f. bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password #Enter node certificate password from step 4.F
  6. Start the Events Service.
Note: If you want to enable TLS for HTTP, do not start the Events Service before enabling TLS for HTTP Interface.

Enable TLS for HTTP Interface

The property ad.es.node.security.tls.http.enabled enables or disables TLS for HTTP communication between Events Service and ElasticSearch.

Note:

Ensure that:

  • ad.es.node.security.basic.auth.enabled=true and ad.es.node.security.tls.transport.enabled=true properties are enabled
  • Stop the Events Service server

before you perform the following steps.

  1. Generate HTTP certificate on the same node where you generated the CA certificate.

    Generate HTTP certificate

    A. export ES_JAVA_HOME=$APPLICATION_HOME/jre
B. cd $APPLICATION_HOME/elasticsearch
C. chmod +x bin/elasticsearch-certutil 
D. bin/elasticsearch-certutil http # This command generates a .zip file that contains certificates and keys to use with Elasticsearch and Kibana. Each folder contains a README.txt explaining how to use these files. 
#Refer : https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html
      1. When asked if you want to generate a CSR, enter n.
      2. When asked if you want to use an existing CA, enter y.
      3. Enter the path to your CA. This is the absolute path to the elastic-stack-ca.p12 file that you generated for your cluster.
      4. Enter the password for your CA.
      5. Enter an expiration value for your certificate. You can enter the validity period in years, months, or days. For example, enter 10000D for 10000D days.
      6. When asked if you want to generate one certificate per node, enter y.Each certificate will have its own private key, and will be issued for a specific hostname or IP address.
      7. When prompted, enter the name of the first node in your cluster. Use the same node name that you used when generating node certificates.
      8. Enter all hostnames (plus 'localhost') used to connect to your first node. These hostnames will be added as DNS names in the Subject Alternative Name (SAN) field in your certificate.List every hostname and variant used to connect to your cluster over HTTPS.
      9. Enter the IP addresses (plus 127.0.0.1) that clients can use to connect to your node.
      10. Repeat these steps for each additional node in your cluster.
E. After generating a certificate for each of your nodes, enter a password for your private key when prompted.  
F. Unzip the generated elasticsearch-ssl-http.zip file. This compressed file contains one directory for both Elasticsearch and Kibana. 
/elasticsearch
|_ README.txt
|_ http.p12
|_ sample-elasticsearch.yml

/kibana
|_ README.txt
|_ elasticsearch-ca.pem
|_ sample-kibana.yml

G. On every node in your cluster, copy the relevant http.p12 certificate to the $APPLICATION_HOME/elasticsearch/config/cert directory.
H.Generate CA cert client-ca.cer, this is needed by events-service
  openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -chain > client-ca.cer
I.On every node in your cluster, copy client-ca.cer at $APPLICATION_HOME/conf/security
  2. Enable TLS on every node by editing the properties as follows:

    On every node

    A. export ES_JAVA_HOME=/home/ec2-user/appdynamics/platform/product/events-service/processor/jre
B. cd $APPLICATION_HOME/elasticsearch
C. chmod +x bin/elasticsearch-keystore
D. bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password #Enter node certificate password from step 4.F
E. bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password #Enter node certificate password from step 4.F

I. vi $APPLICATION_HOME/conf/events-service-api-store.properties 
  - ad.es.node.security.tls.http.enabled=true # Set the property to enable TLS for HTTP
  - ad.es.client.tls.enabled.http.port=true # Enable Java ES client to connect over HTTPS
  3. Start the Events Service.

Verify if TLS is Enabled

Events Service can now communicate to Elastic Search cluster as indicated in the events-service-api-store.log file.

Logs

[2023-03-25T15:11:19,192+05:30]  [INFO ]  [main]  [c.a.a.p.e.n.s.ElasticsearchDependencyModule]  Elasticsearch healthy, it should now be responsive.
[2023-03-25T15:11:19,194+05:30]  [INFO ]  [main]  [c.a.a.p.e.index.MetaDataIndexCreator]  Creating metaData indices...

Verify if TLS is enabled on HTTP as follows:

Verify if TLS is enabled on HTTP

❯ curl -X GET "https://localhost:9200" -k -u elastic:pass123
{
  "name" : "ATKUMAR3-M-2344",
  "cluster_name" : "appdynamics-events-service-cluster",
  "cluster_uuid" : "eLk8ZXUDTe2yKTt-AChDLg",
  "version" : {
    "number" : "8.4.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "42f05b9372a9a4a470db3b52817899b99a76ee73",
    "build_date" : "2022-10-04T07:17:24.662462378Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}