You are here:

Change Failure Prediction Monitoring

Change Failure Prediction Monitoring iCube enables you to create ad-hoc reports that help you understand and evaluate the performance of Change Failure Prediction analytical models over a period of time. You can create reports that can predict failure probability based on risk factors thereby enabling business users to take preemptive action to avoid Change failures. Based on the predictions, you can then monitor the Failure Outcome of closed Changes to understand if the predictions were successful or not.

The following data sets are combined together to create the Change Failure Prediction Monitoring iCube:

ITSA_CFP Monitoring Detail — Provides details about Change Requests along with information regarding the changed risk factors and predicted failure probability.
ITSA_CFP Monitoring Summary — Provides a summary about Change Requests along with prediction error for all closed Change Requests. You can also view the average and median values of prediction error for successful and failed Change Requests.

Change Failure Prediction Monitoring Detail dataset

Attributes

Expand this link to view the list of available attributes and their descriptions:

Attributes	Description
Actual Failure Outcome	Indicates if the Change Request was successful or not. Displays a value of 'Y' for Failed Changed and 'N' for Successful changes. This is applicable only for Closed Changes.
Algorithm Name	Machine Learning (ML) algorithm that was used to predict Change Risk. For example, GBT Binary Classifier, Decision Tree model
Change	Unique Identifier for the Change Request from the source
Change Closed Date	Date on which the Change Request was closed. This is applicable only for closed Changes
Change Request Number	Unique Identifier for the Change Request from the source
Change State	Indicates the current state of the Change Request. For example, New, Scheduled, Authorize, Closed, and so on.
Changed Risk Factor	Name of the Risk Factor that was modified. For example, CI Class, Reschedule Count, and so on. Risk factors are variables that directly contribute to the success or failure of a Change Request.
Deployed Model Version	Version of the ML model that is currently deployed and used for predictions
Failure Probability	Displays the probability of the Change Request failing as of the prediction date This is applicable only for “Open” Changes (Changes where standardized DWH Code = “Open”) For changes in Closed, Canceled, Post-Implementation Review states, Failure Probability will be populated as “Not Applicable”
Last Predicted Failure Probability	Last computed value of Failure Probability for a Change Request as of the prediction date
Last Predicted Failure Risk	Last computed value of Change Failure risk for a Change Request as of the prediction date. The risk of a Change Request failing is calculated based on the failure probability. The risk buckets are configured as follows but can be customized: LOW - Failure Probability <=5% MED - Failure Probability >5% to <=25% MED-HIGH - Failure Probability >25% to <=50% HIGH - Failure Probability >50%
Max Predicted Failure Probability	Maximum value of Failure Probability for a Change Request across all predictions
Min Predicted Failure Probability	Minimum value of Failure Probability for a Change Request across all predictions
New Risk Factor Value	New value of the Risk Factor after it changed on the Prediction date
Old Risk Factor Value	Old value of the Risk Factor before it changed on the Prediction date
Prediction Date	Date and time when the prediction was made

Change Failure Prediction Monitoring Summary dataset

Attributes

Expand this link to view the list of available attributes and their descriptions:

Attributes	Description
Actual Failure Outcome	Indicates if the Change Request was successful or not. Displays a value of 'Y' for Failed Changed and 'N' for Successful changes. This is applicable only for Closed Changes.
Algorithm Name	Machine Learning (ML) algorithm that was used to predict Change Risk. For example, GBT Binary Classifier, Decision Tree model
Change	Unique Identifier for the Change Request from the source
Change Closed Date	Date on which the Change Request was closed. This is applicable only for closed Changes
Change Number	Unique identifier for the Change Request from the source
Deployed Model Version	Version of the ML model that is currently deployed and used for predictions
Last Predicted Failure Probability	Last computed value of Failure Probability for a Change Request as of the prediction date
Last Predicted Failure Risk 1	Last computed value of Change Failure risk for a Change Request as of the prediction date. The risk of a Change Request failing is calculated based on the failure probability. The risk buckets are configured as follows but can be customized: LOW - Failure Probability <=5% MED - Failure Probability >5% to <=25% MED-HIGH - Failure Probability >25% to <=50% HIGH - Failure Probability >50%
Prediction Date	Date and time when the prediction was made

Metrics

Expand this link to view the list of available metrics and their descriptions:

Metric Name	Description	Formula	Expected Value
Average Prediction Error for Failed Changes	Aggregate function to calculate average value of Prediction Error for Change Requests whose Change Failure Outcome is 'Y'	Prediction Error of Failed Changes/Failed Changes	>=0
Average Prediction Error for Successful Changes	Aggregate function to calculate average value of all Prediction Error for Change Requests whose Change Failure Outcome is 'N'	Prediction Error of Successful Changes/Successful Changes	>=0
Closed Changes	Count of all closed changes	Count (Opened Changes) WHERE Change State is 'CLOSED'	>=0
Failed Changes	Count of all changes whose Actual Failure Outcome is 'Y'	Count (Opened Changes) WHERE Actual Failure Outcome is 'Y'	>=0
Median Prediction Error for Failed Changes	Aggregate function to calculate middle value of Prediction Error for Change Requests whose Change Failure Outcome is 'Y'	Median (Prediction Error) of Failed Changes	>=0
Median Prediction Error for Successful Changes	Aggregate function to calculate middle value of Prediction Error for Change Requests whose Change Failure Outcome is 'N'	Median (Prediction Error) of Successful Changes	>=0
Prediction Error	Displays the error in predicting the Failure Probability for every closed Change. For Change Requests whose Change Failure Outcome is Y, the Prediction Error is calculated as (1 - Last Predicted Failure Probability) Else, Prediction Error is the Last Predicted Failure Probability.	Sum (Prediction Error)	>=0
Prediction Error for Failed Changes	Displays the error in predicting the Failure Probability for all failed changes	Sum (Prediction Error) WHERE Actual Failure Outcome is 'Y'	>=0
Prediction Error for Successful Changes	Displays the error in predicting the Failure Probability for all successful changes	Sum (Prediction Error) WHERE Actual Failure Outcome is 'N'	>=0