Retrain a predictive model in ITSI

It is good practice to continuously monitor your incoming data, including historical KPI and service health score values, and retrain a model on newer data if KPIs or entities are added, removed, or changed. For example, if you add an Oracle database to a service, it is best practice to retrain the service's model because the new architecture will create new KPI relationships.

For a model to accurately predict health scores in IT Service Intelligence (ITSI), the data it's making predictions on must have a similar distribution as the data on which the model was trained. Because data distributions are expected to drift over time, deploying a model is not a one-time exercise, but a continuous process.

When to retrain a model

If monitoring your services for changes is too time consuming, a simpler strategy is to train the model periodically. For example, to capture changes to KPIs or service architecture, you might retrain a model every 10 days.

Retrain a service's model in the following situations:

  • You added a new KPI or entity to the service.
  • You removed or changed a KPI or entity in the service.
  • You restored your ITSI configuration (ITSI does not restore MLTK lookup files).
  • You notice that the model's performance is starting to degrade.

Before you retrain a model, test it on recent data to evaluate whether it needs to be retrained.

Prerequisites

  • To retrain a model, the model must be saved in the service definition. For more information, see Train a predictive model in ITSI.
  • Make sure you're viewing the Predictive Analytics tab from the service definition.

Steps

  1. Test the model on recent data:
    1. From the Predictive Analytics tab of the selected service, navigate to the Test a Model section.
    2. Change the test period to a recent time range. Changing the time period retests the model on a recent set of data to determine if it needs to be retrained. You must test on at least 90 minutes of data.

      Note: The appropriate test period varies based on your specific data. For example, if a new KPI was added yesterday, test the model on the last 24 hours. If an outage occurred last week, test it on the last seven days.
    3. Select the model in the Regression Models or Classification Models table to populate the model's metrics.
    4. Analyze the model's metrics. If the metric values have dropped to unacceptable levels for your business, consider retraining the model.
  2. Retrain the model on new data:
    1. Select the same algorithm and algorithm type as you used to train the model. You can modify the time period and training/test split.
    2. Click Train. The existing model is replaced by the retrained model.
    3. Reevaluate the model's metrics to ensure that they are at acceptable levels. For information about evaluating models, see Test a predictive model in ITSI in this manual.
    4. Click Save to save the retrained model into the service definition.

The following diagram illustrates the workflow for retraining a model: RetrainModel.png