Create and Configure Conditions

You can define a condition or a set of conditions to evaluate the performance metrics of your application.

Use the following options to evaluate the conditions:

  • Expression builder embedded in the health rules wizard to create a condition based on a complex expression comprising multiple interdependent metrics.
  • Custom boolean expression to evaluate multiple conditions within a health rule. You can use the AND and OR operators in a boolean expression.

Create a Condition

To create a condition:
  1. In the Critical Condition or Warning Condition window, click + Add Condition to add a new condition component. A row defining the condition is displayed.
  2. Configure the Condition as required.
  3. Continue to add conditions as required. You can add a maximum of 8 conditions. Conditions are designated as A, B, C, and so on.
  4. From the drop-down list above the conditions, select:
    • All: if all of the conditions must evaluate to true to constitute rule violation.
    • Any: if any of the conditions must evaluate to true to constitute rule violation. The health rule evaluates all the specified conditions one by one and if one or more conditions evaluate to true, an alert is triggered. The Health Rule Violation Events Summary displays all the conditions that are violated. For example, you have specified four conditions A, B, C, and D. The health rule evaluates all the four conditions and if condition A and C are violated, an alert is triggered. The Health Rule Violation Events Summary displays the violated conditions A and C that triggered the alert.
    • Custom: if a combination of conditions defined in a boolean expression must evaluate to true to constitute rule violation. For more information on how to create a custom boolean expression, see Create a Custom Boolean expression.
  5. For health rules based on the following health rule types, select:
    • Business Transaction
    • Node health-hardware
    • Node health-transaction performance
    You must specify the evaluation scope in the Critical Criteria and Warning Criteria panels. If you select the percentage of nodes, enter the percentage. If you select the number of nodes, enter an absolute number.

Evaluating Serverless Tiers

When you monitor serverless entities comprising tiers for AWS Lambda, the health rules are evaluated as described below.

Health Rule TypeAffected EntitiesCondition Evaluation CriteriaEvaluation
  • Business transactions
  • Service endpoints
  • Error rates
serverless tier(s)The BT AverageMetrics are aggregated at the tier level.
serverless node(s)
  • Any node
  • % of the Nodes
  • Number of the Nodes
Metrics for serverless tiers are aggregated at the tier level, while the metrics for other tiers are evaluated as per the defined criteria.
Tier/Node Health (Transaction Performance)serverless tier(s)
  • The Tier Average (average for all Nodes in the Tier)
  • Any node
  • % of the Nodes
  • Number of the Nodes
Metrics for serverless tiers are aggregated at the tier level, regardless of the evaluation criteria defined.
serverless node(s)
  • The Tier Average (average for all Nodes in the Tier)
  • Any node
  • % of the Nodes
  • Number of the Nodes
The performance of serverless tiers is not evaluated for Tier/Node Health (Hardware) health rules. AWS does not offer node-level dashboards or metrics because the serverless platform runtime instances spin up and down on-demand.
Tier/Node Health (Hardware)
  • serverless tier(s)
  • serverless node(s)
-The performance of serverless tiers is not evaluated for Tier/Node Health (Hardware) health rules. AWS does not offer node-level dashboards or metrics because the serverless platform runtime instances spin up and down on-demand.

Configure a Condition

To configure a condition:
  1. In the first field of the condition row, enter a name for the condition.This name is used in the generated notification text and in the Splunk AppDynamics console to identify the violation.
    Note: Ensure that you enter a unique name for each condition you define.
  2. From the drop-down list below the Add Condition button, define metrics to evaluate the condition, select:
    • Single metric to specify a single metric:
    1. xyz insert here
    2. From the Value drop-down list, select a qualifier for the metric from the following options:
      Qualifier TypeDescription
      MinimumThe minimum value reported across the configured evaluation time length. Not all metrics have this type.
      MaximumThe maximum value reported across the configured evaluation time length. Not all metrics have this type.
      Note: For the metrics related to Event Storage- Overage Monitoring, the supported qualifier type is Maximum.
      ValueThe arithmetic average of all metric values reported across the configured evaluation time length. This value is based on the type of the metric.
      Note: If you select the time rollup as SUM for custom metrics, the Value is the arithmetic sum of all metric values reported across the configured evaluation time length.
      SumThe sum of all the metric values reported across the configured evaluation time length.
      CountThe number of times the metric value has been measured across the configured evaluation time length.
      Group CountThe number of nodes contributing to a metric value, generally relevant for application or tier level metrics.
      CurrentThe value for the current minute.
    3. To specify a simple metric, click Select a Metric. Metric Selection window is displayed. The metric browser in the Metric Selection window displays metrics appropriate to the health rule type. Alternatively, you can define a relative metric path.
    4. Select a metric to monitor and click Select Metric.
      Note: You can use Alert Sensitivity Tuning to fine-tune metric evaluation for a health rule (that monitors BT, service endpoint or remote service). You must select a single metric to evaluate the condition. See Create a Health Rule and Fine-tune Metric Evaluation for more information.
    • Metric Expression to build a metric expression.
  3. From the drop-down list after the metric, select the type of comparison to evaluate the metric.

    • To limit the effect of the health rule to conditions during which the metric is within a defined range- standard deviations or percentages- from the baseline, select Within Baseline from the menu. To limit the effect of the health rule to when the metric is not within that defined range, select Not Within Baseline. Then select the baseline to use, the numeric qualifier of the unit of evaluation and the unit of evaluation. For example:

      Within Baseline of the Default Baseline by 3 Baseline Standard Deviations

    • To compare the metric with a static literal value, select < Specific value, > Specific Value, = Specific Value, or!= Specific Value. And enter the specific value in the text field. For example:

      Value of Errors per Minute > 100

    • To compare the metric with a baseline, select < Baseline or > Baseline from the drop-down list, and then select the baseline to use, the numeric qualifier of the unit of evaluation and the unit of evaluation. You can use the Baseline Standard Deviation or Baseline Percentage as the unit of evaluation. For example:

      Maximum of Average Response Time is > Baseline of the Daily Trend by 3 Baseline Standard Deviations

      See Dynamic Baselines for information about the baseline options.

      Note: Baseline Percentages: The baseline percentage is the percentage above or below the established baseline at which the condition will trigger. For example, if you have a baseline value of 850 and you have defined a baseline percentage of > 1%, the condition is true if the value is > [850+(850x0.01)] or 859.  To prevent health rule violations from being triggered when the sample sets are too small, these rules are not evaluated if the load- the number of times the value has been measured- is less than 1000. For example, if a very brief time slice is specified, the rule may not violate even if the conditions are met, because the load is not large enough.
  4. If you want to define a 'Persistence Threshold' for the condition to reduce false alerts:
    Note: You can define a persistence threshold for a condition only if you have defined an evaluation time frame of 30 minutes or less.
    1. Select Trigger only when a violation occurs __ times in the last __ min(s).
    2. Define the number of times metric performance data should exceed the defined threshold to constitute a violation.
    3. If required, adjust the evaluation time frame by setting an alternate evaluation time frame.
      Note: You can define a persistence threshold for a condition only if you have defined an evaluation time frame of 30 minutes or less.
  5. If you want to control the health rule status when metric data is unavailable for evaluation, use one of following options for Evaluation for "no data" scenarios:
    • Critical or Warning: The health rule considers this no data scenario as Critical or Warning condition and the health rule status is shown in Red or Yellow respectively.
    • Unknown: The health rule considers the no data scenario as unknown and the health status is shown in Grey.
    • Healthy: The health rule considers the no data scenario as healthy and the heath rule status is shown in Green.
  6. Click Save when done.

Evaluation of Agent Availability Metrics

Using Health Rule Conditions to evaluate agent availability metrics can result in false positives. For example:

  • Agents may not be connecting with controllers due to communication errors for a couple of minutes.
  • Data may be delayed for a couple of minutes due to latency issues.
You can configure the health rules such that you get notified of any occasional one-to-two minute metric loss due to network issues,  or when an agent fails to communicate to the controller.  Configure your Health Rule as follows:
  1. Select Nodes that the health rule affects. You can also set Tiers, however, it is recommended that you set Nodes.
  2. Select Node Health - Hardware, JVM, CLR as the health rule Type.
  3. Use the last five minutes, with a wait time of ten minutes.
  4. Set your condition to be the Sum of < Specific value of three.

This configuration generates a violation when the agent is down for more than two minutes during the last five minutes.

Build an expression

To access the expression builder to create a complex expression as the basis of a condition, select the Metric Expression option from the drop-down list and click Add Expression. The Metric Expression window is displayed that allows you to construct a mathematical expression to use as a metric.

For example, the following expression is created to measure the percent of slow business transactions.

  1. In the Variable Declarationpane of the Mathematical Expression builder, click + Add variable to add a variable.
  2. In the Variable Name field enter a name for the variable.
  3. From the drop-down list, select the qualifier for the metric from the following options:

    Minimum

    The minimum value reported across the configured evaluation time length. This qualifier is not available for all the metrics.

    Maximum

    The maximum value reported across the configured evaluation time length. This qualifier is not available for all the metrics.

    Value

    The arithmetic average of all metric values reported across the configured evaluation time length. This value is based on the type of the metric.

    Note: If you select the time rollup as SUM for custom metrics, the Value is the arithmetic sum of all metric values reported across the configured evaluation time length.

    Sum

    The sum of all the metric values reported across the configured evaluation time length.

    Count

    The number of times the metric value has been measured across the configured evaluation time length.

    Group CountThe number of nodes contributing to a metric value, generally relevant for application or tier level metrics.

    Current

    The value for the current minute.

  4. Click Select a metric to open an embedded metric browser.
    Note:

    Health Rule Evaluation Condition: A health rule is not evaluated if any metric in the expression has a null value. This is to avoid erroneous evaluations as shown in the following examples:

    ExpressionNull ValueEvaluation
    a-b-c a Entire expression is evaluated negative.
    a/bb The number 'a' is divided by zero, evaluates to an error
    a*b a or b Entire expression is evaluated as zero.
  5. Repeat steps 1 through 4 for each metric that you use in the expression.You can remove a variable by clicking the delete icon.
  6. In the Expression pane, build the expression by clicking Insert Variable to insert variables created in the Variable Declaration pane along with appropriate mathematical signs.
  7. When the expression is built, click Save.

Create a Custom Boolean Expression

Once you define all the conditions required for the health rule, you can create a custom boolean expression to evaluate the health rule.

  1. From the drop-down list above the conditions, select Custom option.
  2. Enter a combination of conditions using AND and/or OR operators. For example, (A OR B) AND C.
    Note: Ensure that you enter a valid combination of conditions using AND and OR operators, else the evaluation fails.
  3. Click Save.

Modify the Custom Boolean Expression

  1. Select the expression in the Condition Combination field.
  2. Edit the boolean expression as required.
  3. Click Save.

Delete a Condition

Delete a condition component by clicking the delete (X) icon.

Note: If you delete a condition, update the boolean expression accordingly.