How Text Prediction Worked Before Transformers: n-Gram Counts, Smoothing, Backoff, Perplexity, Autocomplete & Text Generation by Hand

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Count the tokens of a tiny corpus, count how often each word of the vocabulary occurs with COUNTIF, turn the counts into unigram probabilities and measure the average sentence length.
• Write the next token beside every token, count word pairs with COUNTIFS, fill a next-word table for five contexts and check each row total against the context's count.
• Divide each count by its context total to get maximum-likelihood probabilities, multiply them along a sentence with the chain rule, pick the most likely next word with INDEX/MATCH and see a sensible sentence get probability zero.
• Apply add-one smoothing to the next-word table, compare it with a gentler add-k, measure how much probability moves to unseen words and decide which k distorts the counts less.
• Mix bigram and unigram probabilities with four values of λ for a sentence the model has never seen, score each mix with the sum of log-probabilities, pick the best λ and compare it with a simple backoff score.
• Add up log-probabilities of a test sentence under a bigram model and a unigram model, turn them into perplexity with EXP and decide which model predicts the sentence better.
• Turn next-word counts into probabilities, rank the candidates, break a tie with COUNTIF so every word gets its own rank, return the top-3 suggestions with INDEX/MATCH and decide what to show for a context the model has never seen.
• Stack next-word probabilities into cumulative intervals, use given random numbers to pick the next word step by step with a lookup, generate a short sentence from the bigram model and see how this differs from modern models.