AI Fairness & Bias Metrics: Parity, Equal Opportunity & Calibration by Group

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Count TP, FP, FN and TN separately for groups A and B in a test sample of LoanScore, then compare accuracy and base rates.
• Calculate LoanScore's approval rate per group and overall, the demographic parity difference, and chart the rates side by side.
• Compare each group's selection rate with the highest one, flag ratios below the four-fifths threshold and calculate how many approvals a group is short of it.
• Calculate LoanScore's true and false positive rates per group, the equal opportunity difference and the equalised odds difference.
• Check whether LoanScore's scores mean the same in both groups: repay rate per score bin and group, a calibration chart against the mean score, precision (PPV) per group and the predictive parity difference.
• Compute selection rate, TPR and accuracy for each group at four score thresholds, then compare four threshold policies on parity gaps and overall accuracy.
• Split groups A and B by age band, calculate selection rates with a margin of error, flag subgroups too small to judge and compare gaps with and without them.
• Turn the metrics of the previous labs into LoanScore's fairness report card: a Flag/OK status per metric, a count of flags, an overall status and red highlighting for every flag.