Open topic with navigation

Change Failure Prediction dashboard

The Change Failure Prediction dashboard helps you understand and answer questions about changes in your organization. You can monitor changes that are implemented over a period of time and identify factors that contribute to the success or failure of historical changes. This enables you to predict risks associated with implementing current or future changes.

The Change Failure Prediction dashboard helps Change Executives answer the following business questions:

• How many of my planned changes are at risk?
• Which CI class are my high risk changes associated with?
• What is the failure rate of changes that were closed last month?
• How does the failure rate correspond to the failure probability that was predicted last month?

You can use the filters in the dashboard to analyze changes related to specific Departments, Assignment Group Manager Level 1, Primary CI Class, Primary CI, or Change Failure Flag.

Important: In the Change Failure Flag filter, the N flag includes all successful changes, the Y flag includes all unsuccessful changes, and the X flag includes all changes neither successful or unsuccessful. By default, the dashboard is set to include Y and N flags to portray explicit information.

The Change Failure Prediction dashboard consists of the following sections:

• KPIs
  • Closed Changes: Displays the total number of changes that are closed in the last 30 days.
  • Failure Rate: Displays the percentage of closed changes that were unsuccessful in the last 30 days.
  • Success Rate: Displays the year-to-date percentage of changes that have been successfully implemented.
  • Planned Changes: Displays the total number of changes that are planned to start within the next 30 days.
  • Planned Changes at Risk: Displays the number of changes that are planned to start within the next 30 days and whose failure probability is greater than or equal to 25%.
• Change Schedule: This section displays information about changes that are scheduled to be implemented in a given time period as specified by the Planned Start Date filter. The date filter is defaulted to the current date and displays changes that are scheduled for the next 15 days. You can modify the date filter by clicking the icon.

  The table displays the number of changes that are scheduled to start on a specific date, number of changes that are at risk (whose failure probability is greater than or equal to 25%), and indicates the overall risk of deploying the scheduled changes.

  • indicates that less than 25% of the scheduled changes for the specified date are at risk
  • indicates that 25%-50% of the scheduled changes for the specified date are at risk
  • indicates that more than 50% of the scheduled changes for the specified date are at risk

  The specified threshold values are the default configuration provided out-of-the-box and can be customized based on your business requirement.

  You can click the planned changes on a given date to view more information about the changes in the Change Details section.

• Change Details: This section is driven by the Planned Changes section and displays details about Changes, such as description, state, planned start date, failure probability, risk, assignment group, and assigned to. The Failure Probability is generated based on a chosen Change Failure Prediction Machine Learning model and the risk values are configured based on the failure probability. The following values are provided out-of-the-box and can be modified to suit your business needs:
  • LOW: Failure Probability <= 0.05
  • MED: Failure Probability <= 0.25
  • MED-HIGH: Failure Probability <= 0.5
  • HIGH: Failure Probability > 0.5

  You can modify the values by right-clicking the Risk header and selecting Edit.

  You can also select a specific change to view the top risk factors determined by the Change Failure Prediction ML model.

  Risk factors help you in identifying failures associated with a change and enables you to plan the changes for a successful implementation. These failure factors are determined by the ML model based on the out-of-the-box predictor variables, and can be configured to suit your business needs.

• Top Risk Factors: This section is driven by the Change Details section and displays details about Risk factors, such as label name, risk factor details, impact on risk prediction, and risk factor prediction impact.

  For example, for a change_type risk factor, risk factor displays the label name, value displays the risk factor details, impact on risk prediction displays the categorized bucket with colors that signify each threshold range, and the risk factor prediction impact displays the transformed shap value.

  Note: The default format of each Risk factor supported out-of-the box is mentioned in CRP configuration guide.

  Shap values are used to understand the decision made by the ML model that takes the features as input and produces predictions. Predictions are made by the model by quantifying the contribution that each feature brings through Shap.

  The Impact on risk prediction attribute used in this grid is configured as follows, that can be modified to suit your business needs:

  • Major (Red): If the value of Risk Factor Prediction Impact is >= 10%
  • Minor (Amber): If the value of Risk Factor Prediction Impact is < 10% and Risk Factor Proportional Impact is >= 5%
  • Positive (Green): If the value of Risk Factor Prediction Impact is < 10% and Risk Factor Proportional Impact is <= -5%

  Note: Risk Factor Proportional Impact values are used to categorize Risk factors only. If Risk Factor Prediction Impact is < 10%, and Risk Factor Proportional Impact is between -5% and 5%, it is considered Not Significant (filtered out of Top Risk Factors grid).

During ETL, default messages are preconfigured in the Python map (PLP_DWH_D_CHANGE_SUCCESS_MODEL_MONITOR_FORMATTED_RISK_FACTOR_VALUES_PY) and are displayed in the Top Risk Factors grid as follows:

Metrics Used