Evaluating AI & Search: Ranking and LLM Evaluation Metrics

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Score six help-centre queries with precision@k and recall@k, with k kept in one cell so you can change the cut-off, average them over the queries and calculate the hit rate a RAG pipeline cares about.
• Find the rank of the first relevant result for eight queries with MATCH, turn it into a reciprocal rank, average it to the MRR and repeat it with a cut-off (MRR@k).
• Score one ranked result list with graded relevance: discounted gain per rank with LOG(rank + 1, 2), the ideal ranking with LARGE, then DCG@k, IDCG@k and NDCG@k with k in a cell.
• Calculate the precision at every relevant rank for four queries, turn it into average precision (AP), average it to the MAP and see two queries with the same precision@10 but very different AP.
• Sweep the decision threshold of Fern's escalation classifier from 1.0 to 0.0, count true and false positives with COUNTIFS, turn them into TPR and FPR, draw the ROC curve and calculate the AUC with the trapezoid rule.
• Check whether Fern's stated confidence can be trusted: accuracy per confidence bin, the gap to the confidence, the expected and maximum calibration error, and whether the model is over- or underconfident overall.
• Test whether an LLM judge can replace a human grader: a 3 × 3 agreement table with COUNTIFS, the observed agreement, the agreement expected by chance and Cohen's kappa.
• Compare a candidate model with the current one on 240 judged prompts: win rate per category with ties counted half, the pooled win rate, a 95% confidence interval for the decisive win rate and a ship decision, plus a 100% stacked bar chart.