Day 36: CNN architectures: LeNet to ResNet, why skip connections
The lineage that leads to what you actually use
You won't design architectures from scratch — you'll use proven ones. Knowing the lineage tells you *why* the defaults are the defaults. LeNet (1998) proved conv+pool works on digits. AlexNet (2012) scaled it up on GPUs and ignited the deep-learning era. VGG showed depth with simple 3×3 blocks. ResNet (2015) cracked the depth barrier and remains the backbone you'll fine-tune for the Garment Classifier.
The problem ResNet solved
Naively stacking more layers made networks *worse*, not better — gradients vanished on the long path back, and deep plain networks failed to train. ResNet's skip connection (residual connection) adds a layer's input directly to its output: output = F(x) + x. Now gradient has a shortcut straight back through the + x, so even very deep networks train. This one idea — a highway for gradients — recurs everywhere, including the transformer blocks of Stage 3.
Why this matters beyond CNNs
The residual + x pattern is one of the most important ideas in modern deep learning. Transformers (Stage 3) put a skip connection around every attention and feed-forward sub-layer for the exact same reason: without it, deep stacks won't train. Recognizing it in a transformer diagram later will feel like meeting an old friend.
Key terms
- ResNet
- A deep CNN architecture using residual (skip) connections; a standard, reliable backbone for transfer learning.
- Skip / residual connection
- Adding a block's input to its output (F(x) + x), giving gradients a shortcut that makes very deep networks trainable.
- Vanishing gradients
- Gradients shrinking toward zero as they propagate back through many layers, stalling training of deep plain networks.
Why do ResNet skip connections make it possible to train much deeper networks than plain stacked convolutions?
