Attention Scores: Scaling by √d, the Causal Mask & Softmax per Row
This lesson is part of the course
In this lesson
You score every token against every earlier token with q · k ÷ √d_k, block the future with a causal mask and turn each row into attention weights with softmax.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Tokens: Splitting a Prompt, Vocabulary Ids & Token Budgets Locked
- Embeddings: Looking Up Token Vectors & Dot Products Locked
- Positional Signals: Sine & Cosine Waves Added to Embeddings Locked
- Queries, Keys & Values: Weight Tables and SUMPRODUCT Locked
- Attention Scores: Scaling by √d, the Causal Mask & Softmax per Row Locked
- Weighted Values: The Attention Output & Which Token Looks Where Locked
- Next-Token Probabilities: Temperature, Greedy Choice, Top-k & Sampling Locked
- Context Length: Re-Sent History, Turn Costs & Why Long Prompts Weigh More Locked