How Tokenizers Work by Hand: Characters & Pairs, Byte-Pair Merges, Vocabulary Size, Tokens per Word, Sequence Length & Embedding-Table Costs

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Split a toy corpus of six words into one character per cell with MID, count characters per word with LEN, weight everything by how often each word occurs and build the starting vocabulary of five characters.
• List every pair of neighbouring tokens with the & operator, count each pair across the corpus weighted by word frequency, find the most frequent pair with INDEX/MATCH and check the total number of pairs.
• Apply the first learned merge «a» + «t» to every word: mark merged tokens with IF and AND on neighbouring cells, close the gaps with INDEX and SMALL, count tokens per word and see the corpus shrink from 98 to 70 tokens.
• Keep a merge log of four learned merges, build each new token with &, track the vocabulary size and measure how the corpus shrinks in tokens and tokens per word after every merge.
• Tokenize words the tokenizer has never seen: test merge by merge whether a pair occurs with SUMPRODUCT on shifted ranges, replace characters outside the vocabulary by [UNK] with COUNTIF and measure the share of unknown tokens.
• Measure the compression of the toy tokenizer after 0 to 7 merges with tokens per word, characters per token and the tokens saved per merge, and chart how the gains flatten out.
• Compare four fictional vocabulary sizes: embedding parameters and table size grow with the vocabulary, while a bigger vocabulary makes the same document shorter and cuts the attention work.
• Turn word counts into input and output tokens for five toy documents, price each run at fictional rates per 1,000 tokens, add up the monthly cost, check the context limit and chart tokens per word by document type.