How Neural Networks Learn: Gradient Descent, Momentum, RMSProp, Adam, Learning-Rate Schedules & Warm-Up by Hand

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Measure how wrong the Kestrel toy model is with a squared loss, find the slope of that loss with a formula and with finite differences, and read from the sign which way the weight has to move.
• Run ten steps of gradient descent on Kestrel's loss, then compare five learning rates and flag the one that makes training blow up.
• Give Kestrel's two weights a velocity, take the first momentum steps by hand and see the steep direction overshoot while the shallow direction speeds up.
• Keep a running average of squared gradients for two weights, one with big and one with tiny gradients, and see that RMSProp gives both a step of similar size.
• Run five Adam steps on a logged gradient sequence: the two running averages, the bias correction with POWER(β, t) and the final step size.
• See how batch size controls gradient noise, build eight noisy mini-batch gradients from a fixed random column and smooth them with a running average.
• Build three learning-rate schedules for a 1,000-step run – step decay, cosine and warm-up plus cosine – and apply one update with decoupled weight decay.
• Compare four optimisers on the same stretched bowl: the loss after N steps, the first step below a target loss, the leader after N steps and a chart of all four loss curves.