Percent Agreement vs Cohen's Kappa: the Rare-Label Trap
This lesson is part of the course
In this lesson
You calculate percent agreement, chance agreement and Cohen's kappa for two annotator pairs with the same percent agreement, and see why kappa tells them apart.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Label Distribution, Class Balance & Class Weights Locked
- Percent Agreement vs Cohen's Kappa: the Rare-Label Trap Locked
- Fleiss' Kappa: Agreement with More than Two Annotators Locked
- Majority Vote, Label Confidence & Expert Review Locked
- Gold-Standard (Honeypot) Accuracy & Annotator Flags Locked
- Label Noise: What Wrong Test Labels Do to Measured Accuracy (with a Chart) Locked
- Labelling Cost & Throughput: Hours, Review Overhead & Cost per Item Locked
- Active Learning vs Random Sampling: Labels to Target & Budget Saved (with a Chart) Locked