Data Labeling for AI: Inter-Annotator Agreement, Label Quality & Labeling Cost

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Calculate the share of each label, the imbalance ratio, balanced class weights and the accuracy a model would get by always predicting the biggest class.
• Calculate percent agreement, chance agreement and Cohen's kappa for two annotator pairs with the same percent agreement, and see why kappa tells them apart.
• Calculate the agreement on each ticket, the share of each category, the chance agreement and Fleiss' kappa for four annotators.
• Aggregate three annotators' labels by majority vote, count the votes for the winning label, flag the tickets below a confidence threshold and measure the unanimous share.
• Score four annotators on hidden gold items, flag those below the minimum accuracy, total the items to re-check and estimate the wrong labels in the week's output.
• Simulate how symmetric label noise in a test set lowers the measured accuracy of two models and shrinks the gap between them, correct an observed accuracy, and plot both curves.
• Turn items and throughput into annotator and reviewer hours, cost each workflow, calculate the cost per item and the share of the budget spent on review.
• Compare two learning curves, find the first round where each reaches the target accuracy with MINIFS, price the labels saved by active learning and plot both curves.