Adam: First & Second Moments, Bias Correction and the Adam Step
This lesson is part of the course
In this lesson
You run five Adam steps on a logged gradient sequence: the two running averages, the bias correction with POWER(β, t) and the final step size.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Loss & Gradient: Which Way Is Downhill? Locked
- Plain Gradient Descent: Step Tables, Learning Rates & Divergence Locked
- Momentum: A Velocity Term, Overshoot & Faster Progress on a Stretched Bowl Locked
- RMSProp: Running Averages of Squared Gradients & Per-Weight Step Sizes Locked
- Adam: First & Second Moments, Bias Correction and the Adam Step Locked
- Noisy Gradients: Mini-Batch Noise, Batch Size & the Smoothing Effect Locked
- Learning-Rate Schedules: Step Decay, Cosine, Linear Warm-Up & Weight Decay Locked
- The Optimiser Race: Loss after N Steps, Time to Target & Loss Curves Locked