Install the Events Service on Windows

You can install and administer the Events Service on Microsoft Windows systems as a single node or a cluster. Common use cases include:

  • Single-node Events Service— Good for demonstration purposes and other scenarios where data redundancy and high availability are not required.
  • Three-node cluster— The minimum size for a production Events Service cluster.
  • Cluster of four nodes or more— For deployments where increased load or expected sizing exceeds the capacity of a three-node cluster.

Contact Splunk AppDynamics customer support if you anticipate deploying a cluster of 10 or more nodes.

Note: Creating more than one instance of the same service type is not supported on Windows.
Warning: You do not need to specify the installation or data directory for the Events Service installation. If you do, use a different one from the platform directory.

Decide Which Node(s) to Make Master Nodes

Every Events Service node is either a master node or a data node. In an Events Service cluster, the master node both acts as a storage node, and manages the state of the data across the cluster, including the state of the replica. In a single-node deployment, there's not much for the master to do.

The master node is the first node to start up. If the master node becomes unavailable, the worker nodes attempt to elect a new master.

As you install Events Service, you specify the configuration for each node, which consists of two pieces of information:

  • Whether to enable the node to serve as a master, and
  • How many nodes (minimum) must be available in the cluster for a new master to be elected.

This table describes what values to specify as you install:

Order of Node in the Deployment Master-enabled? Minimum available nodes required to elect a new master
First (only) node in a single-node installation true 1
First, second, and third nodes of a three-node cluster true 2
Fourth (or higher) node of a cluster false 2
Warning: Specifying the installation or data directory is optional for the Events Service. If you do this, the directory you specify must not be the platform directory.

Installation Quick Start

Before you can install the Events Service on Windows, you must use Enterprise Console to install the Controller. The EUM Server must be installed separately as it cannot be installed using the Enterprise Console.

Warning: Note that all services on Windows machines must be installed on the Enterprise Console host since the Enterprise Console does not support remote operations on Windows. Therefore, you cannot use the Enterprise Console GUI to deploy an Events Service cluster.
  1. Install the Enterprise Console.
  2. Use the Enterprise Console to create the platform and add hosts.
  3. To install the Controller and Events Service on the same host, use the Express Install option.Use the Custom Install to install a scaled-out Events Service that runs on a host that is separate from the Controller. Custom installations provide more flexibility on where and how to install Controller and Events Service.
  4. (Optional) Install the EUM Server.
  5. Complete the post-install tasks for the Controller, Events Service, and EUM Server.

Deploy a Single-Node Events Service

Note:

Ensure that you have met all of the current Events Service Requirements, including:

  • Have Java Runtime Environment (JRE) version = 1.8
  • Defined Java in the Windows environment variable path, or a JRE folder that is accessible by events-service.exe in the relative path ..\..\jre

To manually install the Events Service on a single node:

  1. Unzip the Events Service distribution archive to a directory on the target host. This creates the events-service directory with the Events Service artifacts.
  2. Begin configuring the connection to the Events Service in the Controller.
    1. With the Controller running, open the Administration Console as the root user.
    2. In the Controller Settings page, search for appdynamics.on.premise.event.service.url.
    3. Replaceconfiguring the connection to the Events Service in the Controller. For example: http://hostname:9080. Select Save.
      Note: If you are putting a load balancer in front of the Events Service (required for a cluster), this will be the VIP for the Events Service as exposed at the load balancer. In this case, it is likely you will need to return to this step after you finish deploying the node and configuring the load balancer.
  3. Obtain the Controller key:
    1. While on the Controller Settingspage, search for the appdynamics.on.premise.event.service.key setting.
      Note: If you do not install the default Events Service, then the appdynamics.on.premise.event.service.key setting is blank. To create the appdynamics.on.premise.event.service.key, use a UUID generator.
    2. Copy the setting's value. You will need this to complete Step 4.
    3. Close the Administration Console.
  4. Configure the connection from the Events Service to the Controller:
    1. From your Windows Explorer, open the events-service\conf\events-service-api-store.properties file to edit.
    2. Locate the controller-key property and add it:
      CODE 
      ad.accountmanager.key.controller=controller-key
  5. Verify the heap settings for the Events Service processes:
    1. Verify that the minimum and maximum heap settings for the two Events Service processes (the Events Service JVM, and the Elasticsearch processes, respectively) are correct and sufficient for your deployment.

      The settings:

      • Are located in the events-service-api-store.properties file.

      • Use g for gigabyte (GB) and m for megabyte (MB).

    2. For the Events Service process, verify:
      CODE 
      ad.jvm.options.name=events-service.vmoptions
ad.jvm.heap.min=1g
ad.jvm.heap.max=1g
    3. For the ElasticSearch process, verify:
      CODE 
      ad.es.jvm.options.name=events-service.vmoptions
ad.es.jvm.heap.min=8g
ad.es.jvm.heap.max=8g
      Note: A production Elasticsearch installation requires 8 GB. For a demonstration installation, you can retain the default of 1 GB.
  6. Save and close the events-service-api-store.properties file.
  7. Install the Events Service as a Windows service, and open the command prompt as an Administrator:
    Note: Do not use PowerShell to do this.
    1. Set the JAVA_HOME environment variable so it specifies your Java installation directory.
      For example: JAVA_HOME=C:\Zulu\zulu-8-jre.
    2. Change the directory to events-service, and then enter:
      CODE 
      bin\events-service.exe service-install -p conf\events-service-api-store.properties --auto-start
      1. This command also installs Elasticsearch (even though it contains no explicit reference to Elasticsearch).
      2. The optional auto-start flag causes the Events Service to be installed as an automatically started service; without this flag, the Events Service is installed as a manually started service.
      3. For verbose installation and operation logging (useful for troubleshooting), include the ‑‑log-verbose flag.
  8. Locate the service name for the Events Service:
    CODE 
    bin\events-service.exe service-list
  9. Open the Windows services and select the Splunk AppDynamicsEvents Service Api Store xxxxx. Select Start.
  10. Check the health of the new node and verify service status:
    1. If "Healthy" appears as the service status, then it indicates that the process is operating normally:
      CODE 
      bin\events-service.exe check-health -hp localhost:9081
    2. For the port, pass the administration port for the Events Service, 9081 by default.
      CODE 
      [appduser@controller-one events-service]$ bin/events-service.exe check-health -hp 192.168.33.22:9081
[2015-12-09T18:30:45,342-08:00] HV000001: Hibernate Validator 5.0.2.Final
[2015-12-09T18:30:45,956-08:00] Individual statuses below:
[2015-12-09T18:30:45,956-08:00] [192.168.33.22:9081] status is [200 OK]
[2015-12-09T18:30:45,956-08:00] Overall status Healthy
...
    3. If the service status does not display as Overall status Healthy, then the service is unhealthy. To correct it, you need to determine the correct key mappings between the following Events Service configuration and the Controller settings:
      1. appdynamics.es.eum.key should map to ad.accountmanager.key.eum
      2. appdynamics.on.premise.event.service.key should map to ad.accountmanager.key.controller
      3. appdynamics.on.premise.event.service.key should map to ad.accountmanager.key.mds
      4. If the values of the key mappings are blank in the Admin Console, then use a UUID generator to create them.
      5. Use a UUID generator to create a value for ad.accountmanager.key.ops.
      6. Use a UUID generator to create a different value for the following keys (you can use the same UUID for all three keys):
        1. ad.accountmanager.key.slm
        2. ad.accountmanager.key.jf
        3. ad.accountmanager.key.service
  11. Configure the connections from the Analytics Agent, EUM Server, or Database Monitoring agents to the Events Service, as described in Connect to the Events Service.

Deploy an Events Service Cluster (Three Nodes)

The following steps describe how to deploy an Events Service cluster made up of three nodes, the minimum size of the Events Service cluster. These steps apply whether you are performing a new installation of a cluster or expanding a single node deployment into a three-node cluster. For information on expanding beyond a three-node cluster, see Adding Nodes to a Cluster.

Warning: Note that all services on Windows machines must be installed on the Enterprise Console host since the Enterprise Console does not support remote operations on Windows. Therefore, you cannot use the Enterprise Console GUI to deploy an Events Service cluster.

Before starting, review the topology notes in Events Service Deployment and make sure that all machines in the cluster meet the system requirements.

When ready, perform these steps on each of the three nodes.

  1. Follow the steps for configuring a single node cluster in the 1-node installation above. Additionally, configure the following settings in the conf\events-service-api-store.properties file:
    1. Change the value of the ad.es.node.minimum_master_nodes property to 2:
      CODE 
      ad.es.node.minimum_master_nodes=2

      The setting specifies the minimum number of master-eligible instances that must be available in order to elect a new master. Since an Events Service cluster has three master nodes, this value should be two for a cluster.

    2. Set the value of ad.es.event.index.shards to the number of nodes, in this case, three:
      CODE 
      ad.es.event.index.shards=3

      You do not need to change this value if it is already higher than the number of nodes.

    3. Set the replication factor to 1 by changing the ad.es.event.index.replicas and ad.es.metadata.replicas properties, as follows:
      CODE 
      ad.es.event.index.replicas=1
ad.es.event.index.hotLifespanDays=10
ad.es.metadata.replicas=1
    4. For the unicast hosts property, add the hostname or IP address, along with the port 9300, for each node in the cluster:
      CODE 
      ad.es.node.unicast.hosts=node1.example.com:9300,node2.example.com:9300,node3.example.com:9300
    5. Change the publish host to the IP address or hostname of this machine. For example:
      CODE 
      ad.es.node.network.publish.host=node2.example.com
    6. Configure heap space for the Events Service and ElasticSearch processes, as follows:
      1. To set the Events Service process heap size to 1 GB, for example, use the following properties:
        CODE 
        ad.jvm.options.name=events-service.vmoptions
ad.jvm.heap.min=1g
ad.jvm.heap.max=1g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB).

      2. For the ElasticSearch process, the heap size should be set to half the size of the available RAM on the system, up to a maximum of 31 GB. To set the ElasticSearch process heap size to 8 GB, for example, set the properties as follows:
        CODE 
        ad.es.jvm.options.name=events-service.vmoptions
ad.es.jvm.heap.min=8g
ad.es.jvm.heap.max=8g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB).

    7. Save and close the file.
  2. Install the Events Service as a Windows service:
    CODE 
    bin\events-service.exe service-install -p conf\events-service-api-store.properties  --auto-start

    The optional auto-start flag causes the Events Service to be installed as an automatically started service. If you do not include the flag, the Events Service is installed as a manually started service. An additional option, ‑‑log-verbose, increases the verbosity of installation and operation logging, which is useful for troubleshooting.

  3. Enter the following command to find the service name for the Events Service:
    CODE 
    bin\events-service.exe service-list
  4. Pass the service name returned by the service-list command as the -s parameter argument in the following command:
    PYTHON 
    bin\events-service.exe service-start -s "<Name from service-list>"

    Be sure to enclose the name in double-quotes.

  5. Check the health of the new node using the following command. At least two nodes must be running before you run the command.
    CODE 
    bin\events-service.exe check-health -hp localhost:9081

    For the port, pass the administration port for the Events Service, 9081 by default. Verify that "Healthy" appears as the service status, indicating that the process is operating normally:

    CODE 
    [appduser@controller-one events-service]$ bin/events-service.exe check-health -hp 192.168.33.22:9081
[2015-12-09T18:30:45,342-08:00] HV000001: Hibernate Validator 5.0.2.Final
[2015-12-09T18:30:45,956-08:00] Individual statuses below:
[2015-12-09T18:30:45,956-08:00] [192.168.33.22:9081] status is [200 OK]
[2015-12-09T18:30:45,956-08:00] Overall status Healthy
...
  6. Configure a load balancer to distribute traffic to the Events Service cluster, as described in Load Balance Events Service Traffic.
  7. Connect the Controller and other clients—Analytics Agent, EUM Server, or Database Monitoring agents—to the Events Service, as described in Connect to the Events Service.

Expand an Events Service Cluster

The Events Service cluster is horizontally scalable. You can add nodes to an existing cluster without affecting or having to restart the existing nodes.

Warning: Note that all services on Windows machines must be installed on the Enterprise Console host since the Enterprise Console does not support remote operations on Windows. Therefore, you cannot use the Enterprise Console GUI to expand an Events Service cluster.

Before starting, prepare the new cluster machine. Verify system requirements and prepare the environment as described above.

For each node beyond the original three master nodes, download and configure the nodes as previously described. The configuration steps for any nodes added to the cluster after the initial three master nodes are as follows:

  1. For each cluster nodes beyond the initial three master nodes, open the conf\events-service-api-store.properties for editing and make these configuration changes:
    1. Set the ad.es.node.master value to false:
      CODE 
      ad.es.node.master=false
    2. Set the ad.es.node.minimum_master_nodes value to 2.
      CODE 
      ad.es.node.minimum_master_nodes=2
    3. Set the value of ad.es.event.index.shards to the number of nodes in the cluster. You do not need to change this value if it is already higher than the number of nodes.
      CODE 
      ad.es.event.index.shards=<number_of_nodes>

      You do not need to change this value if it is already higher than the number of nodes.

    4. For the unicast hosts property, add the hostnames or IP addresses of all nodes in the cluster, including the node you are adding. For each node specify the ports on which the nodes communicate, 9300-9400. For example:
      CODE 
      ad.es.node.unicast.hosts=node1.example.com[9300-9400],node2.example.com[9300-9400],node3.example.com[9300-9400],node4.example.com[9300-9400]

      You do not need to reconfigure the unicast hosts settings for existing cluster members, as the new node can join the cluster dynamically.

    5. Change the publish host to the IP address or hostname of this machine. For example:
      CODE 
      ad.es.node.network.publish.host=node4.example.com
    6. Configure heap space for the Events Service and ElasticSearch processes, as follows:
      1. To set the Events Service process heap size to 1 GB, for example, use the following properties:
        CODE 
        ad.jvm.options.name=events-service.vmoptions
ad.jvm.heap.min=1g
ad.jvm.heap.max=1g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB).

      2. For the ElasticSearch process, the heap size should be set to half the size of the available RAM on the system, up to a maximum of 31 GB. To set the ElasticSearch process heap size to 8 GB, set the properties as follows:
        CODE 
        ad.es.jvm.options.name=events-service.vmoptions
ad.es.jvm.heap.min=8g
ad.es.jvm.heap.max=8g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB).

    7. Save and close the file.
  2. Install the Events Service as a Windows service:
    CODE 
    bin\events-service.exe service-install -p conf\events-service-api-store.properties --auto-start

    The optional auto-start flag causes the Events Service to be installed as an automatically started service. If you do not include the flag, the Events Service is installed as a manually started service. An additional option, ‑‑log-verbose, increases the verbosity of installation and operation logging, which is useful for troubleshooting.

  3. Enter the following command to find the service name for the Events Service:
    CODE 
    bin\events-service.exe service-list
  4. Pass the service name returned by the service-list command as the -s parameter argument in the following command:
    PYTHON 
    bin\events-service.exe service-start -s "<Name from service-list>"
  5. Check the health of the new node:
    CODE 
    bin\events-service.exe check-health -hp localhost:9081
    Note: At least two nodes must be running before you run the command.

    For the port, pass the administration port for the Events Service, 9081 by default. Verify that "Healthy" appears as the service status, indicating that the process is operating normally:

    CODE 
    [appduser@controller-one events-service]$ bin/events-service.bin check-health -hp 192.168.33.22:9081
[2015-12-09T18:30:45,342-08:00] HV000001: Hibernate Validator 5.0.2.Final
[2015-12-09T18:30:45,956-08:00] Individual statuses below:
[2015-12-09T18:30:45,956-08:00] [192.168.33.22:9081] status is [200 OK]
[2015-12-09T18:30:45,956-08:00] Overall status Healthy
...
  6. Modify your load balancer rules to include the new cluster node. For more information, see Load Balance Events Service Traffic.

Start and Stop the Events Service

At installation, the Events Service is installed as a service and is left running upon completion of the installation. You can stop and stop it as a service or as a foreground process, as described here or by using the GUI.

Start and Stop as a Foreground Process

To start the Events Service as a foreground process, however, use the following command:

CODE 
bin\events-service.exe start -p conf\events-service-api-store.properties

To stop the Events Service as a foreground process, use this command:

CODE 
bin\events-service.exe stop

Stop and Start as a Windows Service

You can stop and start the Events Service as a Windows service from the Services Manager. You can also stop and start it using the events-service.exe tool, as here:

  1. Enter the following command to find the service name for the Events Service:

    CODE 
    bin\events-service.exe service-list

  2. Pass the service name returned by the service-list command as the -s parameter argument to the following command. Enclose the service name in double-quotes.

    PYTHON 
    bin\events-service.exe service-start -s "<Name from service-list>"

To stop the service, run this command:

PYTHON 
bin\events-service.exe service-stop -s "<Name from service-list>"

Remove a Node

To remove a node that is not enabled for operation as a master node from the cluster, simply stop the Events Services on the node or remove the machine it runs on from the network.

Note the following guidelines:

  • You cannot remove nodes such that the resulting cluster size is two.
  • A cluster that consists of three or more nodes can't be reduced in size to a single node Events Service.

After you remove a node, be sure to adjust your load balancer rules to remove the old cluster member. See Load Balance Events Service Traffic for more information.

If you are not using a load balancer with a cluster deployment, keep in mind that the connection settings for the first master node that reports to the Controller at installation time are written to the Controller setting that identifies the Events Service to the Controller. If you remove a master node in that case, check whether the removed master node is node identified as the Events Service destination URL in the Controller connection settings; adjust the setting if so. See Connect to the Events Service for more information.

To reconfigure an existing node to enable operation as a master node, or add a new node with the master option enabled:

CODE 
ad.es.node.master=true

If reconfiguring a node, restart the node after changing the configuration.