Learning-Rate Schedules: Step Decay, Cosine, Linear Warm-Up & Weight Decay
This lesson is part of the course
In this lesson
You build three learning-rate schedules for a 1,000-step run – step decay, cosine and warm-up plus cosine – and apply one update with decoupled weight decay.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Loss & Gradient: Which Way Is Downhill? Locked
- Plain Gradient Descent: Step Tables, Learning Rates & Divergence Locked
- Momentum: A Velocity Term, Overshoot & Faster Progress on a Stretched Bowl Locked
- RMSProp: Running Averages of Squared Gradients & Per-Weight Step Sizes Locked
- Adam: First & Second Moments, Bias Correction and the Adam Step Locked
- Noisy Gradients: Mini-Batch Noise, Batch Size & the Smoothing Effect Locked
- Learning-Rate Schedules: Step Decay, Cosine, Linear Warm-Up & Weight Decay Locked
- The Optimiser Race: Loss after N Steps, Time to Target & Loss Curves Locked