Build a Monitoring Extension Using Java

You can create Java monitoring extensions that enable the Machine Agent and Server Visibility to collect custom metrics, which you define and provide, and report them to the Controller. This is an alternative to adding monitoring extensions using scripts.

When you capture custom metrics with a monitoring extension, they are supported by the same Splunk AppDynamics services that you receive for the standard metrics captured with the Splunk AppDynamics application and Machine Agents. These services include automatic baselining, anomaly detection, display in the Metric Browser, availability for display on custom dashboards, and availability for use in policies to trigger alerts and other actions.

This page describes how to create a monitoring extension in Java.

To the Agent, a monitoring extension is a task that runs on a fixed schedule and collects metrics.

Before You Begin

Before creating your own extension from scratch, review the extensions on theSplunk AppDynamics community. See GitHub for a free download of the extensions

New extensions are constantly being added. It is possible that someone has already created what you need, or something close enough, that you can download it and use it after making a few modifications.

Create a Monitoring Extension in Java

To create a monitoring extension in Java:

  1. Create your extension class. See Create the Monitoring Extension.
  2. Create a monitor.xml configuration file. See Create the monitor.xml File.
  3. Create a subdirectory (<your_extension_dir>) under <agent_home>/monitors .
  4. Place the extension class file and themonitor.xml file (plus any dependent jar files) in <your_extension_dir>.
  5. Enter the Controller access information and credentials. See Configure the Machine Agent.
    Note: Make sure that your controller-info and command line parameters are correctly configured. Required properties include: Controller name, port number, and account access key.
  6. Restart the Machine Agent.

Create the Monitoring Extension Class

Create a monitoring extension class by extending the AManagedMonitor com.singularity.ee.agent.systemagent.api MachineAgent.jar

The monitoring extension class performs these tasks:

  • Populates a hash map with the values of the metrics that you want to add to Splunk AppDynamics. You obtain these metrics based on your environment and the source from which you derive your custom metrics.
  • Defines the type of metrics to collect using the MetricWriter class. See Metric Processing Qualifier.
  • Uploads the metrics to the Controller using the execute() method of the AManagedMonitor class.

See Extension_Class_Source.txt for example source code.

Metric Path

All custom metrics processed by the Machine Agent and Server Visibility appear in the Metric Browser > Application Infrastructure Performance. Use the "I" character to specify the path from Application Infrastructure Performance to the custom metric. If the metrics apply to a specific tier, use the metric path for the tier, with "Component" followed by a colon ":" and the tier name or tier ID.

For example to associate a metric with tier AccountService, you would specify the metric path as: Server|Component:AccountService|Custom Metrics|Path|

The metric would then appear under the tree in the Metric Browser:

Metric Browser Tree

Warning: You can report a custom metric only to the tier that is associated with the Machine Agent. If you attempt to publish metrics to a different tier, the metrics are not reported.

You can insert a custom metric next to an existing type of metric. For example, the following declaration causes the custom metric named pool usage to appear next to the JMX metrics: name="Server|Component:<tier-name>|JMX|Pool|First|pool usage",value=10

You can then use the metric in health rules similar to other types of JMX metrics.

Tip: To test the appearance of your custom metric in the Controller API. post the metric data to the Machine Agent's REST API. Pass the path, name type and values of the metric as URL arguments. See Machine Agent HTTP Listener.

Metric Names

Metric names must be unique within the same metric path but need not be unique for the entire metric hierarchy.

We recommend that you use short metric names so that they are visible when they are displayed in the Metric Browser.

Prepend the metric path to the metric name when you upload the metrics to the Controller.

aggregator="aggregator type" METRIC_AGGREGATION_TYPE_AVERAGE METRIC_AGGREGATION_TYPE_SUM METRIC_AGGREGATION_TYPE_OBSERVATION METRIC_TIME_ROLLUP_TYPE_AVERAGE METRIC_TIME_ROLLUP_TYPE_SUM METRIC_TIME_ROLLUP_TYPE_CURRENT METRIC_CLUSTER_ROLLUP_TYPE_INDIVIDUAL METRIC_CLUSTER_ROLLUP_TYPE_COLLECTIVE

Metric Processing Qualifier

The Controller has various qualifiers for how it processes a metric regarding aggregation, time roll-up, and tier roll-up.

There are three types of metric qualifiers:

  • Aggregator qualifier
  • Time roll-up qualifier
  • Cluster roll-up qualifier

You specify these options with the enumerated types provided by the MetricWriter class.

Aggregation Qualifier

Theaggregatorqualifier specifies how the Machine Agent aggregates the values reported during a one-minute period.

Specify the aggregation qualifier as

This value is an enumerated type. Valid values are:

Aggregator Type Description

METRIC_AGGREGATION_TYPE_AVERAGE

Default. Average of all reported values in that minute.

METRIC_AGGREGATION_TYPE_SUM

Sum of all reported values in that minute, causes the metric to behave like a counter.

METRIC_AGGREGATION_TYPE_OBSERVATION

Last reported value in the minute. If no value is reported in that minute, the last reported value is used.

Time Roll-Up

Thetime-rollupqualifier specifies how the Controller rolls up the values when it converts from one-minute granularity tables to 10-minute granularity tables, and 60-minute granularity tables over time.

Roll-Up Strategy Description

METRIC_TIME_ROLLUP_TYPE_AVERAGE

Average of all one-minute data points when adding it to the 10-minute or 60-minute granularity table.

METRIC_TIME_ROLLUP_TYPE_SUM

Sum of all one-minute data points when adding it to the 10-minute or 60-minute granularity table.

METRIC_TIME_ROLLUP_TYPE_CURRENT

Last reported one-minute data point in that 10-minute or 60-minute interval.

Cluster Roll-Up

Thecluster-rollupqualifier specifies how the Controller aggregates metric values in a tier.

Roll-up Strategy Description

METRIC_CLUSTER_ROLLUP_TYPE_INDIVIDUAL

Aggregates the metric value by averaging the metric values across each node in the tier.

METRIC_CLUSTER_ROLLUP_TYPE_COLLECTIVE

Aggregates the metric value by adding up the metric values for all the nodes in the tier.

For example, if a tier has two nodes, Node A and Node B, and Node A has three errors per minute and Node B has seven errors per minute, the INDIVIDUAL qualifier reports a value of five errors per minute and COLLECTIVE qualifier reports ten errors per minute. INDIVIDUAL is appropriate for metrics such as % CPU Busy where you want the value for each node. COLLECTIVE is appropriate for metrics such as Number of Calls where you want a value for the entire tier.

Sample Monitoring Extension Class

The NGinXMonitor

  • Active Connections: number of active connections
  • Accepts: number of accepted requests
  • Handled: number of handled requests
  • Requests: total number of requests
  • Reading: number of reads
  • Writing: number of writes
  • Waiting: number of keep-alive connections

The source for the extension class is included as an attachment.

Create monitor.xml

Create a monitor.xml <monitor>

  1. Set the <name> to the name of your Java monitoring extension class.
  2. Set the <type> to managed .
  3. The <execution-style> can be continuous or periodic.
    • Continuous means to collect the metrics averaged over time; for example, average CPU usage per minute. In continuous execution, the Machine Agent invokes the extension once and the program runs continuously, returning data every 60 seconds.

    • Periodic means to invoke the monitor at a specified frequency. In periodic execution, the Machine Agent invokes the extension, runs it briefly, and returns the data on the schedule set by the <execution-frequency-in-seconds>

      Note: Do not set the <execution-frequency-in-seconds> greater than 300 seconds (five minutes). The extension must collect metrics at least once every five minutes.
  4. If you select periodic for the execution style, set the frequency of collection in <execution-frequency-in-seconds> element. The default frequency is 60 seconds. If you select continuous , this setting is ignored.
  5. Set the <type> in the <monitor-run-task> child element to java .
  6. Set the <execution-timeout-in-secs> to the number of seconds before the extension times out.
  7. Specify any task arguments required by the Java-based extension in the <task-arguments> name default-value is_required is_required <task-arguments>
    <argument name="password" is-required="true" default-value="welcome" />
  8. Set the <classpath> to the jar file that contains your extension's classes. Include any dependent jar files, separated by semicolons.
  9. Set the<impl-class> to the full path of the class that the Machine Agent invokes.

Sample monitor.xml Files

This attached monitor.xml file configures the NGinXMonitor monitoring extension. This extension executes every 60 seconds.

This jar attached monitor.xml file configures the MysqlMonitor. This monitor executes every 60 seconds, has four required task arguments and one optional task argument and one dependent