Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size
This lesson is part of the course
In this lesson
You compare four fictional vocabulary sizes: embedding parameters and table size grow with the vocabulary, while a bigger vocabulary makes the same document shorter and cuts the attention work.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Characters and Frequencies: A Toy Word List, Characters per Word and the Starting Vocabulary Locked
- Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency Locked
- The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps Locked
- Several Merges: A Merge Table, the Vocabulary After Each Step and Tokens per Word Shrinking Locked
- Tokenizing New Words: Applying the Merges in Order, Unknown Characters and Fallbacks Locked
- Compression: Tokens per Word, Characters per Token and the Effect of More Merges Locked
- Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size Locked
- Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart Locked