Cross-Validation & Hyperparameter Tuning by Hand: Train/Validation/Test Splits, k-Fold Scores, Grid Search, Overfitting Curves & Data Leakage

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Assign 20 days to train, validation and test with MOD on a given shuffle number, count the days per split and compare the share of busy days in each split with the whole dataset.
• Deal the 16 pool days into 4 folds with MOD, find the fold that ends up without a single busy day and fix it with stratified folds built from a running count within each class.
• Measure the distance from a held-out day to the 12 days of the other folds, rank the neighbours, average the rentals of the k nearest and score the prediction against the actual value.
• Turn 16 out-of-fold predictions into absolute errors, score every fold with its mean absolute error, average the four fold scores into the cross-validation estimate and measure how much they spread.
• Average a grid of six k values × five fold scores, add a standard error to every row, find the setting with the lowest mean error and apply the one-standard-error rule to pick a simpler, equally good setting.
• Compare training and validation error for trees of depth 1 to 8, compute the gap, label every depth as underfitting, good fit or overfitting, find the sweet spot and draw both curves in one line chart.
• Scale temperatures with minimum and maximum taken from the training rows only, flag records that appear in two folds and check a forecasting rule month by month in time order instead of with shuffled folds.
• Average the fold scores of four candidate models, pick the winner by its CV MAE, score the retrained model once on the four locked test days and chart the fold scores for the report.