Calibration: Does 90% Confident Mean 90% Correct?
This lesson is part of the course
In this lesson
You check whether Fern's stated confidence can be trusted: accuracy per confidence bin, the gap to the confidence, the expected and maximum calibration error, and whether the model is over- or underconfident overall.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Search Quality: Precision@k, Recall@k & Hit Rate Locked
- Mean Reciprocal Rank: How Soon Is the First Good Answer? Locked
- NDCG@k: Graded Relevance and the Ideal Ranking Locked
- Average Precision & MAP: Rewarding Good Order Locked
- Thresholds, the ROC Curve & AUC by Trapezoids Locked
- Calibration: Does 90% Confident Mean 90% Correct? Locked
- LLM-as-a-Judge: Percent Agreement & Cohen's Kappa Locked
- A/B Testing Two Models: Win Rate, Ties & a Confidence Interval Locked