How Language Models Work by Hand: Tokens, Embeddings, Attention Weights, Softmax & Sampling with Temperature

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Turn a split prompt into token ids with a lookup that handles an unknown word, measure tokens per word, compare a character rule of thumb with the real count and price the prompt for a day of traffic.
• Look up a 4-number vector for each token of the prompt, compare tokens with dot products and count the parameters of an embedding table.
• Build sine and cosine position signals for four positions, see how the wavelength grows with the dimension pair and add the signals to the token embeddings.
• Turn each token's input vector into a query, a key and a value with three small weight tables and SUMPRODUCT, and count the weights of a larger attention layer.
• Score every token against every earlier token with q · k ÷ √d_k, block the future with a causal mask and turn each row into attention weights with softmax.
• Mix the value vectors with the attention weights into the output of the attention layer, find which earlier token "boxes" attends to most and chart its weights.
• Turn eight logits into next-token probabilities at a temperature, pick the greedy token, find the top-k cut-off and sample a token with a fixed random number.
• Work out how many input tokens each turn of a chat re-sends, price a turn, see the n × n score grid grow and estimate the memory for the stored keys and values.