Day 95: Multi-head attention & the transformer block
Many attentions at once, then the full block
One attention computes one kind of relationship. Multi-head attention runs several attention operations in parallel — each 'head' with its own Q/K/V projections — so the model attends to different relationships simultaneously (one head might track syntax, another subject-object links). Their outputs concatenate and mix. This is the attention sub-layer of the transformer block.
The transformer block
A block stacks: multi-head attention → add & normalize (a skip connection, Stage 1 Day 36, plus layer norm) → a position-wise feed-forward network → add & normalize again. Stack N of these blocks and you have a GPT. Every ingredient is something you know: attention (Day 94), skip connections, MLPs (Stage 1). Recognizing the residual + x around each sub-layer is meeting an old friend.
Depth = stacked blocks
A '32-layer' model has 32 of these blocks stacked. The residual connections (why they're essential: Stage 1 Day 36) are what make stacking 32+ blocks trainable — without them, gradients vanish through the depth. This is the single idea carrying from CNNs to transformers, and naming it is a strong interview signal.
Key terms
- Multi-head attention
- Running several attention operations in parallel, each capturing different relationships, then combining their outputs.
- Transformer block
- Multi-head attention + feed-forward network, each wrapped in a residual connection and layer normalization.
- Layer normalization
- Normalizing activations within each token to stabilize training in deep transformer stacks.
What is the purpose of using multiple attention heads instead of one?
