Noisy Gradients: Mini-Batch Noise, Batch Size & the Smoothing Effect
This lesson is part of the course
In this lesson
You see how batch size controls gradient noise, build eight noisy mini-batch gradients from a fixed random column and smooth them with a running average.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Loss & Gradient: Which Way Is Downhill? Locked
- Plain Gradient Descent: Step Tables, Learning Rates & Divergence Locked
- Momentum: A Velocity Term, Overshoot & Faster Progress on a Stretched Bowl Locked
- RMSProp: Running Averages of Squared Gradients & Per-Weight Step Sizes Locked
- Adam: First & Second Moments, Bias Correction and the Adam Step Locked
- Noisy Gradients: Mini-Batch Noise, Batch Size & the Smoothing Effect Locked
- Learning-Rate Schedules: Step Decay, Cosine, Linear Warm-Up & Weight Decay Locked
- The Optimiser Race: Loss after N Steps, Time to Target & Loss Curves Locked