Day 91: Why transformers: attention as the core idea
The most leveraged week of the roadmap
Stage 3 builds the AI Stylist — an agent that answers 'formal shirts under ₹2500 in Medium, matching black trousers' by calling your real inventory API, retrieving product knowledge, and reasoning about it, streamed token by token. But the roadmap is emphatic that the *transformer* week is the highest-leverage stretch of the whole 230 days. Understand attention deeply and every LLM decision downstream — quantization, serving, context windows, agents — rests on solid ground instead of hand-waving.
What attention replaced
Before transformers, sequence models (RNNs, LSTMs) processed text one token at a time, carrying a hidden state forward. This was inherently sequential (slow to train) and struggled to connect distant words. Attention lets every position directly look at every other position in parallel and weigh how relevant each is — 'trousers' can attend strongly to 'black' and 'matching' regardless of distance. Parallel, and good at long-range dependencies: the two things RNNs lacked.
You already know the key operation
Attention scores are dot products (Stage 0 Day 7) — 'how aligned is this query with that key?'. And every transformer block wraps its sub-layers in skip connections (Stage 1 Day 36). The transformer isn't a wall of new ideas; it's a clever arrangement of operations you've already met. That's why building nanoGPT this week (Days 96–97) will feel like assembly, not revelation.
Key terms
- Attention
- A mechanism letting each sequence position weigh the relevance of every other position and aggregate their information.
- Transformer
- The architecture built on attention that underpins modern LLMs; parallelizable and strong at long-range dependencies.
- Long-range dependency
- A relationship between distant tokens in a sequence, which attention captures directly and RNNs struggled with.
What are the two key advantages of attention over the RNNs it replaced?
