Training Compute: The 6 × Parameters × Tokens Rule & Scientific Notation
This lesson is part of the course
In this lesson
You estimate the training compute of four fictional model sizes with the 6 × parameters × tokens rule of thumb, write the huge results in scientific notation and compare the plan with a tokens-per-parameter heuristic.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Dataset Tokens: Documents, Words, Deduplication & the Validation Split Locked
- Training Compute: The 6 × Parameters × Tokens Rule & Scientific Notation Locked
- GPU-Hours, Utilisation & Cluster Size for a Deadline Locked
- On-Demand, Reserved or Interruptible: Pricing the GPU-Hours Locked
- Fine-Tuning Budgets: Epochs, Low-Rank Adapters & Memory per Parameter Locked
- Experiment Grids: Learning Rates × Batch Sizes × Seeds and Early Stopping Locked
- Scaling Curves: Fitting a Power Law and Predicting Loss at a Bigger Budget Locked
- The Project Budget: Checkpoints, Storage, Evaluations, Contingency & the Chart Locked