Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart
This lesson is part of the course
In this lesson
You turn word counts into input and output tokens for five toy documents, price each run at fictional rates per 1,000 tokens, add up the monthly cost, check the context limit and chart tokens per word by document type.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Characters and Frequencies: A Toy Word List, Characters per Word and the Starting Vocabulary Locked
- Pair Counts: Adjacent Pairs per Word, One Token per Cell, Weighted by Word Frequency Locked
- The First Merge: Rewriting Every Word with IF on Neighbouring Cells and Closing the Gaps Locked
- Several Merges: A Merge Table, the Vocabulary After Each Step and Tokens per Word Shrinking Locked
- Tokenizing New Words: Applying the Merges in Order, Unknown Characters and Fallbacks Locked
- Compression: Tokens per Word, Characters per Token and the Effect of More Merges Locked
- Vocabulary Trade-offs: Vocabulary × Embedding Width = Parameters, Sequence Length vs Vocabulary Size Locked
- Token Budgets & Dashboard: Tokens per Toy Document, Cost at a Fictional Price per 1,000 Tokens and a Chart Locked