Plain Gradient Descent: Step Tables, Learning Rates & Divergence
This lesson is part of the course
In this lesson
You run ten steps of gradient descent on Kestrel's loss, then compare five learning rates and flag the one that makes training blow up.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Loss & Gradient: Which Way Is Downhill? Locked
- Plain Gradient Descent: Step Tables, Learning Rates & Divergence Locked
- Momentum: A Velocity Term, Overshoot & Faster Progress on a Stretched Bowl Locked
- RMSProp: Running Averages of Squared Gradients & Per-Weight Step Sizes Locked
- Adam: First & Second Moments, Bias Correction and the Adam Step Locked
- Noisy Gradients: Mini-Batch Noise, Batch Size & the Smoothing Effect Locked
- Learning-Rate Schedules: Step Decay, Cosine, Linear Warm-Up & Weight Decay Locked
- The Optimiser Race: Loss after N Steps, Time to Target & Loss Curves Locked