Weighted Values: The Attention Output & Which Token Looks Where
This lesson is part of the course
In this lesson
You mix the value vectors with the attention weights into the output of the attention layer, find which earlier token "boxes" attends to most and chart its weights.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Tokens: Splitting a Prompt, Vocabulary Ids & Token Budgets Locked
- Embeddings: Looking Up Token Vectors & Dot Products Locked
- Positional Signals: Sine & Cosine Waves Added to Embeddings Locked
- Queries, Keys & Values: Weight Tables and SUMPRODUCT Locked
- Attention Scores: Scaling by √d, the Causal Mask & Softmax per Row Locked
- Weighted Values: The Attention Output & Which Token Looks Where Locked
- Next-Token Probabilities: Temperature, Greedy Choice, Top-k & Sampling Locked
- Context Length: Re-Sent History, Turn Costs & Why Long Prompts Weigh More Locked