LLM-as-a-Judge: Percent Agreement & Cohen's Kappa
This lesson is part of the course
In this lesson
You test whether an LLM judge can replace a human grader: a 3 × 3 agreement table with COUNTIFS, the observed agreement, the agreement expected by chance and Cohen's kappa.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Search Quality: Precision@k, Recall@k & Hit Rate Locked
- Mean Reciprocal Rank: How Soon Is the First Good Answer? Locked
- NDCG@k: Graded Relevance and the Ideal Ranking Locked
- Average Precision & MAP: Rewarding Good Order Locked
- Thresholds, the ROC Curve & AUC by Trapezoids Locked
- Calibration: Does 90% Confident Mean 90% Correct? Locked
- LLM-as-a-Judge: Percent Agreement & Cohen's Kappa Locked
- A/B Testing Two Models: Win Rate, Ties & a Confidence Interval Locked