A/B Testing Two Models: Win Rate, Ties & a Confidence Interval
This lesson is part of the course
In this lesson
You compare a candidate model with the current one on 240 judged prompts: win rate per category with ties counted half, the pooled win rate, a 95% confidence interval for the decisive win rate and a ship decision, plus a 100% stacked bar chart.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Search Quality: Precision@k, Recall@k & Hit Rate Locked
- Mean Reciprocal Rank: How Soon Is the First Good Answer? Locked
- NDCG@k: Graded Relevance and the Ideal Ranking Locked
- Average Precision & MAP: Rewarding Good Order Locked
- Thresholds, the ROC Curve & AUC by Trapezoids Locked
- Calibration: Does 90% Confident Mean 90% Correct? Locked
- LLM-as-a-Judge: Percent Agreement & Cohen's Kappa Locked
- A/B Testing Two Models: Win Rate, Ties & a Confidence Interval Locked