Configure external Elasticsearch

Configure Splunk AppDynamics Virtual Appliance to use self-managed Elasticsearch for Events Service data.

Use self-managed Elasticsearch 8.19.11 with the AppDynamics plugin build for the Elasticsearch maintenance version.

  1. Prepare a Elasticsearch service.
    • Provide a stable private endpoint reachable from OpenShift.
    • Enable Elasticsearch basic authentication.
    • Enable TLS for the HTTP endpoint and use a certificate whose SAN covers the configured DNS name or IP address.
    • Configure production capacity, high availability, monitoring, snapshots, restore, and disaster recovery.

    A single-node deployment is suitable only for testing. Use a multi-node design for production.

  2. Install the supported Elasticsearch version when using the Linux TAR file.
    JSON 
    ES_VERSION=8.19.11
mkdir -p ~/elastic
cd ~/elastic
curl -L -O "https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz"
curl -L -O "https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz.sha512"
sha512sum -c "elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz.sha512"
tar -xzf "elasticsearch-${ES_VERSION}-linux-x86_64.tar.gz"
cd "elasticsearch-${ES_VERSION}"
./bin/elasticsearch --version
  3. Optional: Create a default Elasticsearch user.
    CODE 
    ./bin/elasticsearch-reset-password -u elastic -b --url https://localhost:9200

    This creates a default user. You can also create a custom user and supply credentials in the later steps.

  4. Configure Elasticsearch TLS.
    CODE 
    # 1. Generate a Certificate Authority (CA). --pem gives you plain files instead of a
#    PKCS#12 keystore, which is what BYOK's config/es-ca.crt convention expects.
./bin/elasticsearch-certutil ca --pem --out ca.zip --days 3650
unzip ca.zip -d ca
# -> ca/ca/ca.crt   (public CA certificate — this is what you give to BYOK)
# -> ca/ca/ca.key   (CA private key — keep this safe, never give it to BYOK/customers)

# 2. Generate the ES server's own certificate, signed by that CA. The --dns value MUST
#    be the exact hostname events/eum will use to connect (this becomes the cert's SAN —
#    Subject Alternative Name). Add --ip if you also need to connect by IP for testing.
./bin/elasticsearch-certutil cert \
  --ca-cert ca/ca/ca.crt --ca-key ca/ca/ca.key \
  --pem --days 3650 --name es-node \
  --dns "${ES_HOST}" \
  --out es-node-cert.zip
unzip es-node-cert.zip -d node
# -> node/es-node/es-node.crt   (server certificate)
# -> node/es-node/es-node.key   (server private key)

# 3. Verify the SAN is exactly what you expect BEFORE configuring Elasticsearch —
openssl x509 -in node/es-node/es-node.crt -noout -text | grep -A1 "Subject Alternative Name"
    Sample Output:
    CODE 
    X509v3 Subject Alternative Name:
                DNS:elasticsearch.example.internal

    This command uses the Elasticsearch utility to generate certificates for TLS. However, you can generate your own certificates.

  5. Edit the config/elasticsearch.yml file and append the below information to capture certificate details.
    CODE 
    cluster.name: appd-external-es
node.name: es-node-0

path.data: /home/ec2-user/elastic/elasticsearch-8.19.11/data
path.logs: /home/ec2-user/elastic/elasticsearch-8.19.11/logs

network.host: 0.0.0.0
http.port: 9200

discovery.type: single-node   # single-node test only; remove for a real multi-node cluster

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: certs/es-node.crt
xpack.security.http.ssl.key: certs/es-node.key
xpack.security.http.ssl.certificate_authorities: certs/ca.crt

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.certificate: certs/es-node.crt
xpack.security.transport.ssl.key: certs/es-node.key
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.verification_mode: certificate
  6. Configure Elasticsearch TLS and authentication.

    Configure TLS for the HTTP and transport interfaces. Issue the HTTP certificate for the exact host name that Virtual Appliance workloads use.

    Create a supported Elasticsearch user and password for Virtual Appliance. The source configuration uses the superuser role.

    Verify the endpoint without bypassing TLS checks:

    CODE 
    curl --cacert <path-to-ca> -u <username>:<password> \
  "https://<elasticsearch-host>:<port>/_cluster/health?pretty"
  7. Start Elasticsearch.
    CODE 
    # Foreground (for this first test — Ctrl+C stops it):
./bin/elasticsearch

# Or as a background/production service on EC2, once you're confident it's correct:
# nohup ./bin/elasticsearch > es.log 2>&1 &
# (For a real deployment, install as a systemd service instead — see the Elastic docs
#  for "Install Elasticsearch as a service".)
  8. Install the four AppDynamics Elasticsearch plugins on every node.

    Use byok8s/images/es-plugins-<elasticsearch-version>.tar.gz from the Virtual Appliance bundle. The archive contains these plugin directories:

    • meter-store-upsert-script-non-native
    • account-config-upsert-script-non-native
    • merge-upsert-script-non-native
    • expression-script-non-native
    Extract the archive, copy the directories into the Elasticsearch plugins directory, and restart every node.
    Attention: Do not use elasticsearch-plugin install for these already-extracted directories.
    CODE 
    tar -xzf es-plugins-<elasticsearch-version>.tar.gz
cp -a plugins/* <elasticsearch-home>/plugins/
curl --cacert <path-to-ca> -u <username>:<password> \
  "https://<elasticsearch-host>:<port>/_cat/plugins?v"

    Confirm that all four plugins appear on every node. A mismatched plugin can prevent Elasticsearch from starting or executing the required scripts.

  9. Configure the Elasticsearch endpoint in config/external-elasticsearch.yaml.
    CODE 
    statefulServices:
  elasticsearch:
    schemaVersion: 1
    external:
      host: "elasticsearch.example.internal"
      port: 9200
      useHostname: true

    Set useHostname: true when the certificate identifies the endpoint by DNS name. Use false only when the certificate contains the configured IP address as an IP SAN.

  10. Add the Elasticsearch CA to the installation bundle.

    For a private or provider CA that is not in the system trust store, save the public PEM CA chain as config/es-ca.crt.

  11. Add the Elasticsearch credential to the protected config/secrets.yaml file.
    CODE 
    elasticsearch:
  external:
    username: "<elasticsearch-username>"
    password: "<elasticsearch-password>"
    roles: superuser
  12. Validate Elasticsearch before installation.
    • Confirm that cluster health is green or an understood yellow state.
      CODE 
      curl --cacert <ca> -u <user>:<pass> <https://<your-es-host>>:<port>/_cluster/health
  13. Verify Elasticsearch after installation.
    • Run the cluster-health request from a workload pod and confirm that it returns without TLS or authentication errors.
    • Check Events Service logs for script-execution failures that indicate a missing or incompatible plugin.
Install Virtual Appliance with external services after you configure every external service.