A/B Testing & Experimentation

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Write Harbor & Pine's first test plan: baselines for the primary, secondary and guardrail metric, the target conversion rate from the MDE and the guardrail limit.
• Calculate how many visitors each variant of the delivery-date test needs to detect an 8% lift with 95% confidence and 80% power.
• Turn the sample size into a test duration from real weekday traffic, round it to full weeks and check it against the maximum length.
• Analyse the finished delivery-date test: difference in conversion, z-score, two-sided p-value and the 95% confidence interval of the difference.
• Test whether the delivery-date variant also raised revenue per visitor with a Welch t-test, then write the ship decision from the test plan.
• Test four homepage headlines against the control, see how fast false winners pile up and protect the result with a Bonferroni correction.
• Check whether the 50/50 split of the new checkout test really came out 50/50, with a chi-square goodness-of-fit test and a strict SRM threshold.
• Replay 20 A/A tests that were checked four times each, measure how peeking inflates the false positive rate and fix it with a Pocock boundary.