Day 93: Positional encoding: why order needs to be injected
Attention is blind to order — so we tell it
Attention treats its inputs as a *set* — it computes all-to-all relationships with no inherent notion of which token came first. But 'dog bites man' and 'man bites dog' mean different things. Positional encoding injects order information into the token embeddings, so the model knows each token's position. Without it, a transformer literally cannot tell word order.
How position gets encoded
- Sinusoidal (original transformer) — fixed patterns of sines/cosines added to embeddings; no parameters, extrapolates somewhat to longer sequences.
- Learned — a trainable vector per position; simple, but limited to trained lengths.
- Rotary (RoPE) — rotates query/key vectors by position; the modern standard (Llama, Qwen) because it handles relative positions and longer contexts gracefully.
Positional encoding and context length are linked
How a model encodes position bounds how far it can extend its context window. RoPE's popularity is partly because techniques exist to *stretch* it to longer contexts than trained on. When you later choose a model by its context window (Day 100), the positional encoding is quietly the reason some models extend cleanly and others don't.
Key terms
- Positional encoding
- Information added to token representations so the order-blind attention mechanism knows each token's position.
- RoPE (Rotary Position Embedding)
- A modern positional method rotating query/key vectors by position; handles relative positions and long contexts well.
Why do transformers need positional encoding?
