Scaling Curves: Fitting a Power Law and Predicting Loss at a Bigger Budget
This lesson is part of the course
In this lesson
You take six fictional pilot runs, fit a power law between compute and validation loss with LN and SLOPE, predict the loss of the big run with FORECAST.LINEAR and EXP, and chart the curve.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Dataset Tokens: Documents, Words, Deduplication & the Validation Split Locked
- Training Compute: The 6 × Parameters × Tokens Rule & Scientific Notation Locked
- GPU-Hours, Utilisation & Cluster Size for a Deadline Locked
- On-Demand, Reserved or Interruptible: Pricing the GPU-Hours Locked
- Fine-Tuning Budgets: Epochs, Low-Rank Adapters & Memory per Parameter Locked
- Experiment Grids: Learning Rates × Batch Sizes × Seeds and Early Stopping Locked
- Scaling Curves: Fitting a Power Law and Predicting Loss at a Bigger Budget Locked
- The Project Budget: Checkpoints, Storage, Evaluations, Contingency & the Chart Locked