Compression: Tokens per Word, Characters per Token and the Effect of More Merges
This lesson is part of the course
In this lesson
You measure the compression of the toy tokenizer after 0 to 7 merges with tokens per word, characters per token and the tokens saved per merge, and chart how the gains flatten out.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Characters and Frequencies: A Toy Word List, Characters per Word and the Starting Vocabulary Locked
- Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency Locked
- The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps Locked
- Several Merges: A Merge Table, the Vocabulary After Each Step and Tokens per Word Shrinking Locked
- Tokenizing New Words: Applying the Merges in Order, Unknown Characters and Fallbacks Locked
- Compression: Tokens per Word, Characters per Token and the Effect of More Merges Locked
- Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size Locked
- Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart Locked