Positional Signals: Sine & Cosine Waves Added to Embeddings
This lesson is part of the course
In this lesson
You build sine and cosine position signals for four positions, see how the wavelength grows with the dimension pair and add the signals to the token embeddings.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Tokens: Splitting a Prompt, Vocabulary Ids & Token Budgets Locked
- Embeddings: Looking Up Token Vectors & Dot Products Locked
- Positional Signals: Sine & Cosine Waves Added to Embeddings Locked
- Queries, Keys & Values: Weight Tables and SUMPRODUCT Locked
- Attention Scores: Scaling by √d, the Causal Mask & Softmax per Row Locked
- Weighted Values: The Attention Output & Which Token Looks Where Locked
- Next-Token Probabilities: Temperature, Greedy Choice, Top-k & Sampling Locked
- Context Length: Re-Sent History, Turn Costs & Why Long Prompts Weigh More Locked