The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps
This lesson is part of the course
In this lesson
You apply the first learned merge «a» + «t» to every word: mark merged tokens with IF and AND on neighbouring cells, close the gaps with INDEX and SMALL, count tokens per word and see the corpus shrink from 98 to 70 tokens.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Characters and Frequencies: A Toy Word List, Characters per Word and the Starting Vocabulary Locked
- Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency Locked
- The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps Locked
- Several Merges: A Merge Table, the Vocabulary After Each Step and Tokens per Word Shrinking Locked
- Tokenizing New Words: Applying the Merges in Order, Unknown Characters and Fallbacks Locked
- Compression: Tokens per Word, Characters per Token and the Effect of More Merges Locked
- Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size Locked
- Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart Locked