ML Model Monitoring: Data Drift, PSI, KS Test & Performance Decay

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Bin the debt-to-income ratio of Corvel Lending's training data and of this week's applications with COUNTIFS and FREQUENCY, and turn the counts into shares you can compare.
• Calculate the PSI of credit utilisation bin by bin, add it up and classify six features as stable, moderate shift or major shift with the usual 0.1 / 0.25 rule of thumb.
• Build the cumulative distributions of applicant income for the baseline and the current month, find the KS statistic as the largest vertical gap, compare it with the critical value and chart both curves.
• Test whether the mix of application channels has shifted with a chi-square goodness-of-fit test, measure the size of the shift with the Jensen–Shannon divergence and find the channel that gained most.
• Measure how the model's predicted default probabilities shift from January to June with a prediction PSI per month, find the first month that reaches the watch level and chart the band mix.
• Calculate precision, recall and balanced accuracy for six monthly cohorts whose default labels have arrived, and measure how far June has fallen below the validation result.
• Turn eight monitoring metrics into Green / Amber / Red statuses with amber and red thresholds, derive one action for the on-call team and colour the dashboard with conditional formatting.
• Put a dollar figure on the drift (excess credit losses), compare it with the one-off cost of retraining and let a payback rule decide: retrain now or keep monitoring.