Install the Events Service on Linux

This page describes how to install and administer the Events Service on Linux systems through the CLI. Steps for scaling up an embedded Events Service using the Enterprise Console are also included.

The Enterprise Console automates the task of installing and administering an Events Service deployment through either the GUI or CLI. For information on installing Events Service using the Enterprise Console, see Custom Install.

Warning: You do not need to specify the installation or data directory for the Events Service installation. If you do, use a different one from the platform directory.

Events Service Host Requirements

Before starting, be sure to review the Release Notes for known issues and late-breaking information on using the Events Service and Enterprise Console. Also, observe the following requirements:

  • The Events Service can be deployed as a single node or as a multi-node cluster of 3 or more nodes.

  • The versions of Linux supported include the flavors and versions supported by the Controller, as indicated by Prepare Linux for the Controller.

  • The Events Service must run on a dedicated machine. The machine should not run other applications or processes not related to the Events Service.

  • Use appropriately sized hardware for the Events Service machines. The Enterprise Console checks the target system for minimum hardware requirements. For more information on these requirements, see the description of the profile argument to the Events Service install command in Install the Events Service Cluster.
  • The Controller and Events Service must reside on the same local network and communicate by the internal network. Do not deploy the cluster to nodes on different networks, whether relative to each other or to the Controller where the Enterprise Console runs. When identifying cluster hosts in the configuration, you will need to use the internal DNS name or IP address of the host, not the externally routable DNS name. For example, in terms of an AWS deployment, use the private IP address such as 172.31.2.19 rather than public DNS hostname such as ec2-34-201-129-89.us-west-2.compute.amazonaws. com.

  • Make sure that the appropriate ports on each Events Service host are open. See Port Settingsfor more information.

  • The Enterprise Console uses an SSH key to access the Events Services hosts. See the section below for information on generating the key.

  • Events Service nodes normally operate behind a load balancer. When installing an Events Service node, the Enterprise Console automatically configures a direct connection from the Controller to the node. If you deploy a cluster, the first primary node is automatically configured as the connection point in the Controller. You will need to reconfigure the Controller to connect through the load balancer VIP after installation, as described below. For sample configurations, see Load Balance Events Service Traffic.

Note: We recommend you that to grant execute permissions to the /tmp directory, otherwise the installation fails with an error. However, to troubleshoot this error, see

Error if the /tmp Directory is Missing the Execute Permission

.

Port Settings

Each machine must have the following ports accessible to external (outside the cluster) traffic:

  • Events Service API Store Port: 9080
  • Events Service API Store Admin Port: 9081

For a cluster, ensure that the following ports are open for communication between machines within the cluster. Typically, this requires configuring iptables or OS-level firewall software on each machine to open the ports listed

  • 9300 – 9400

The following shows an example of iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 9080 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9081 -j ACCEPT
-A INPUT -m state --state NEW -m multiport -p tcp --dports 9300:9400 -j ACCEPT

If a port on the Events Service node is blocked, the Events Service installation command will fail for the node and the Enterprise Console command output and logs will include an error message similar to the following:

failed on host: <ip_address> with message: Uri [http://localhost:9080/_ping] is un-pingable.

If you see this error, make sure that the ports indicated in this section are available to other cluster nodes.

Create the SSH Key

When installing Events Service, you will need to provide the SSH key that the Enterprise Console can use to access Events Service hosts remotely. Before starting, create the PEM public and private keys in RSA format. The key file must not use password protection.

For example, using ssh-keygen, you can create the key using the following command:

ssh-keygen -t rsa -b 2048 -v -m pem

Configure SSH Passwordless Login

The Enterprise Console needs to be able to access each cluster machine using passwordless SSH. Before starting, enable key-based SSH access.

This setup involves generating a key pair on the Enterprise Console host and adding the Enterprise Console's public key as an authorized key on the cluster nodes. The following steps take you through the configuration procedure for an example scenario. You will need to adjust the steps based on your environment.

If you are using EC2 instances on AWS, the following steps are taken care of for you when you provision the EC2 hosts. At that time, you are prompted for your PEM file, which causes the public key for the PEM file to be copied to the authorized_keys

On the Enterprise Console machine, follow these steps:

  1. Log in to the Enterprise Console machine or switch to the user you will use to perform the deployment:
    su - $USER
  2. Create a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows: 
    mkdir -p ~/.ssh
chmod 700 ~/.ssh
  3. Change to the directory: 
    cd .ssh
  4. Generate PEM public and private keys in RSA format:
    ssh-keygen -t rsa -b 2048 -v -m pem
    The key file must not use password protection.
  5. Enter a name for the file in which to save the key when prompted, such as appd-analytics .
  6. Rename the key file by adding the .pem extension: 
    mv appd-analytics appd-analytics.pem
    You will later configure the path to it as the sshKeyFile setting in the Enterprise Console configuration file, as described in Deploying an Events Service Cluster.
  7. Transfer a copy of the public key to the cluster machines. For example, you can use scp to perform the transfer as follows: 
    scp ~/.ssh/myserver.pub host1:/tmp
scp ~/.ssh/myserver.pub host2:/tmp
scp ~/.ssh/myserver.pub host3:/tmp
    Continuing with the example, myserver should be appd-analytics. The first time you connect you may need to confirm the connection to add the cluster machine to the list of known hosts and enter the user's password.
  8. On each cluster node (host1, host2, and host3), create the .ssh directory in the user home directory, if not already there, and add the public key you just copied as an authorized key:
    cat /tmp/appd-analytics.pub >> .ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
  9. Test the configuration from the Controller machine by trying to log in to a cluster node by ssh:
    ssh host1
    Note: If unable to connect, make sure that the cluster machines have the openssh-server package installed and that you have modified the operating system firewall rules to accept SSH connections. If successful, you can use the Enterprise Console on the Controller host to deploy the Events Service cluster, as described next.
If the Enterprise Console attempts to install the Events Service on a node for which passwordless SSH is not properly configured, you will see the following error message: 
./bin/platform-admin.sh add-hosts --hosts <es_host_1> <es_host_2> <es_host_3> --credential <credential name> --platform-name <name of platform>
...
Failed to connect to the remote host. Please verify that the host name and credentials you provided are correct.
For more information, consult the documentation: https://docs.appdynamics.com/display/PRO45/Administer+the+Enterprise+Console#AdministertheEnterpriseConsole-manage-hostsManageHosts

If you encounter this error, use the instructions in this section to double-check your passwordless SSH configuration. Also, you need to and add-hosts first then install event-service. If SSH configuration is not setup correctly, add-hosts commands will fail.

Warning: The add-hosts command is to add hostnames into platforms. During the events-service installation, the hosts come from the platform hosts, and then they are used on the events-service.

Tune the Operating System for Production Cluster Nodes

Before installing the Events Service cluster, you need to perform a few manual changes as described below. These are particularly relevant for production Events Service deployments. On each node in the cluster, make these configuration changes:

  1. Using a text editor, open /etc/sysctl.conf and add the following:
    vm.max_map_count=262144
  2. Raise the open file descriptor limit in /etc/security/limits.conf, as follows: 
    <username_running_eventsservice>     soft    nofile          96000
<username_running_eventsservice>     hard    nofile          96000
<username_running_eventsservice>     soft    memlock         unlimited
<username_running_eventsservice>     hard    memlock         unlimited
  3. Disable swap memory by running the following command. Remove swap mount points by removing or commenting the lines in /etc/fstab that contain the word swap. 
    swapoff -a

Installing the Events Service Using the GUI

In the GUI, the Express Install option automatically installs an Events Service on the same host as the Controller. The Custom Install option can install an embedded or scaled-up Events Service. If you install an embedded Events Service and want to switch to a scaled-up Events Service, complete the steps described in Scaling Up an Embedded Events Service.

Installing the Events Service Using the CLI

  1. Set up load balancing. See Load Balance Events Service Traffic for information about configuring the load balancer.
  2. At the command line, navigate to the platform-admin Install the Enterprise Console.
  3. If it has been more than one day since your last session, you will have to log in with the following command:
    bin/platform-admin.sh login --user-name <admin_username> --password <admin_password>
  4. Create a platform as follows:
    bin/platform-admin.sh create-platform --name <platform_name> --installation-dir <platform_installation_directory>
    The installation directory is the directory where the Enterprise Console installs all platform components.
    Note: The same installation directory should exist and is used on all remote nodes. This is done to maintain the homogeneity of the configuration across different nodes.
  5. Add the SSH key that the Enterprise Console will use to access and manage the Events Service hosts remotely. (See<file path to the key file>The platform-name Create the SSH Key for more information): 
    bin/platform-admin.sh add-credential --credential-name <name> --type ssh --user-name <username> --ssh-key-file <file path to the key file> --platform-name <name of platform>
    <file path to the key file> is the private key for the Enterprise Console machine. The installation process uses the keys to connect to the Events Service hosts. The keys are not deployed, but instead, encrypted and stored in the Enterprise Console database. The platform-name parameter is optional.
  6. Add hosts to the platform, passing the credential you added to the platform:
    bin/platform-admin.sh add-hosts --hosts es_host_1 es_host_2 es_host_3 --credential <credential name> --platform-name <name of platform>
    The platform-name parameter is optional.
  7. On each Events Service destination node in the cluster, create an installation directory for the Events Service. This is the directory you specified as the installation-dir argument while creating the platform in step (2).
  8. Again at the command line for the Enterprise Console machine, run the following command from the platform-admin directory. Pass the cluster configuration settings as arguments to the command. The format for the command is the following: 
    bin/platform-admin.sh submit-job --platform-name <platform-name> --service events-service --job install --target-version <latest> --args profile=<dev> serviceActionHost=<es_host_1> serviceActionHost=<es_host_2> serviceActionHost=<es_host_3>

    The platform-name and jvmTempDir parameters are optional.

    Arguments are:

    • jvmTempDir: Use this argument to override default JVM temporary /tmp directory in Linux installations.

    • hosts: Use this argument or host-file to specify the internal DNS hostnames or IP addresses of the cluster hosts in your deployment. With this argument, pass the hostnames or addresses as parameters. For example:

      --hosts 192.168.32.105 192.168.32.106 192.168.32.107

    • host-file: As an alternative to specifying hosts as --hosts arguments, pass them as a list in a text file you specify with this argument. Specify the internal DNS hostname or IP address for each cluster host as a separate line in the plain text file:

      192.168.32.105
192.168.32.106
192.168.32.107

    • profile: By default (with profile not specified), the installation is considered a production installation. Specifying a developer profile (‑‑profile dev) directs the Enterprise Console to use a reduced hardware profile requirement, suitable for non-production environments only. The Enterprise Console checks for the following resources:

      • For a dev profile, 1 core CPU, 1 GB RAM, and 2 GB disk space.

      • Otherwise, 4 core CPU, 12 GB RAM, and 128 GB of disk space.

    For example: 
    bin/platform-admin.sh add-hosts --hosts ip-172-31-20-21.us-west-2.compute.internal ip-172-31-20-22.us-west-2.compute.internal ip-172-31-20-23.us-west-2.compute.internal
    If using a hosts text file, use the following command: 
    bin/platform-admin.sh add-hosts --hosts --host-file=/home/appduser/hosts.txt
  9. Log in to each Events Service node machine, and run the script for setting up the environment as follows:
    Note:
    • The tune-system.sh script is used for optimizing your environment. It is optional.
    • Enable the following property:
    INSTALL_BOOTSTRAP_MASTER_ES8=true
    1. Add execute permission to the tune-system.sh script:
      chmod +x tune-system.sh
./tune-system.sh
    2. Run the script: 
      sudo <installation_dir>/events-service/processor/bin/tool/tune-system.sh
  10. If you are using a load balancer, use the virtual IP for the Events Service as presented at the load balancer. Configure the Controller connection to the Events Service as follows:
    1. Open the Administration Console.
    2. In the Controller settings pane, find appdynamics.on.premise.event.service.url and change its value to the URL of the virtual IP for the Events Service at the load balancer.
    It may take a few minutes for the Controller and Events Service to synchronize account information after you modify connection settings in the console.
When finished, use the Enterprise Console for any Events Service administrative functions. You should not need to access the cluster node machines directly once they are deployed. In particular, do not attempt to use scripts included in the Events Service node home directories.The Enterprise Console automatically updates the Controller configurations after installation.

Scaling Up an Embedded Events Service

The following steps describe how to scale up an Events Service that is on a shared host with the Controller. This allows the embedded Events Service to run on a separate host. You can also install the Events Service on a separate host directly by using the Custom Install.

  1. Set up load balancing. See Load Balance Events Service Traffic for information about configuring the load balancer.
  2. Open the Enterprise Console GUI.
  3. Verify that the credentials and hosts you want to use are added to the Splunk AppDynamics platform. For more information, see Administer the Enterprise Console.
    1. On the Credential page, add the SSH credentials for the hosts on which you want to install the Events Service.
    2. On the Hosts page, add the hosts. The Enterprise Console uses these hosts for the scaled-up Events Service, which requires at least one host or three or more hosts.
  4. On the Events Servicepage, navigate to More link and select the Events Service and click Scale Up Events Service under More and complete the wizard. When you enter hosts to use for a scaled-up Events Service, do not include the Controller host.
    Note: You do not need to restart the Controller since that is automatically done for you by the scale-up job.
  5. Log in to each node machine, and run the script for setting up the environment as follows: 
    sudo <installation_dir>/events-service/latest/bin/tool/tune-system.sh
  6. Navigate to the Controller page.
  7. By default, the Enterprise Console configures the Events Service connection in the Controller to refer to the first primary node defined in the cluster. If you are using a load balancer, as recommended, you need to change this Controller setting to point to the Events Service VIP as presented at the load balancer instead, as follows:
    1. Open the Administration Console.
    2. In the Controller settings pane, find appdynamics.on.premise.event.service.url.
    3. Change the value of the setting for the URL to the VIP for the Events Service at the load balancer.
  8. By default, Database Monitoring stores events data in the Events Service embedded in the Controller. To have it use the Events Service you just deployed, ensure that the appdynamics.on.premise.event.service.key value matches with ad.accountmanager.key.controller value inside the events-service-api-store.properties file.
    Note: Note that only newly generated Database Monitoring data will be stored in the Events Service; previously collected data will remain in the embedded Events Service instance unless it is migrated to the new Events Service. See

    Connect to the Events Service

    .
  9. It may take a few minutes for the Controller and Events Service to synchronize account information after you modify connection settings in the Enterprise Console.

Troubleshooting Installation

If the Enterprise Console crashes or shuts down while installing the Events Service, the GUI may display that the installation is in progress. To resolve this issue, uninstall the Events Service with the CLI. Then, install the Events Service with the CLI.

Handling ClusterFormationFailureHelper Error

When you run new Events Service 23.2.0 on a single node event service cluster, the following error may be displayed.

[WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node.

To resolve this issue:

  1. Uninstall the Events Service from Enterprise Console.
  2. Ensure that you have tuned the operating system for cluster nodes as defined in this section.
  3. Re-install the Events Service.

Error if the /tmp Directory is Missing the Execute Permission

If the /tmp directory is missing execute permission, the Events Service installation or upgrade fails with the following errors:

failed to map segment from shared object or
failed to allocate closure

  1. Run the following command to verify the execute permission of the folder:

    mount | grep '/tmp'
/dev/sda1 on /tmp type ext4 (rw,noexec)

  2. Check the privileges for the following folders:

    mount | grep '/var/tmp'
mount | grep 'home'

You can troubleshoot the error by:

  • Exporting the files to the Events Service directory.
  • Removing the noexec mount option for the required directories.

Exporting the Files to the Events Service Directory

  1. Remove the Events Service files that were added during the failed installation.

  2. Locate the binaries of Events Service in the following path:

    cd /opt/appdynamics/platform/platform-admin/archives/events-service/<version>cd /opt/appdynamics/platform/platform-admin/archives/events-service/<version>

  3. Unzip events-service.zip

    unzip events-service.zip

  4. Rename the events-service.zip to events-services.zip.original.

    mv events-service.zip events-service.zip.original

  5. In the extracted events-service directory, add the following lines to both the events-service.vmoptions and events-service-elasticsearch.vmoptions files:

    -Djna.tmpdir=/opt/appdynamics/platform/product/events-service/tmp
-Djansi.tmpdir=/opt/appdynamics/platform/product/events-service/tmp

  6. Archive the events-service folder with the modified .vmoptions files:

    zip -r events-service.zip events-service

  7. Create a temporary directory in the events-service folder where .vmoptions files is mentioned.

    mkdir -p /opt/appdynamics/platform/product/events-service/tmp

  8. Export the temporary directory using the following command:

    export TMPDIR=<path/to/es_tmp>
  9. Install the Events Service using Enterprise Console.

Removing thenoexecMount Option for the Required Directories

Remove the noexec Mount Option for the Required Directories
sudo umount /tmp
sudo mount -o remount,rw /tmp