Day 23: Building micrograd: backprop through a 2-layer net
Topological backprop and a real neuron
Yesterday each operation knew how to push gradient one step back. Today you orchestrate them: to backprop the whole graph, set the output's grad to 1.0, then call every node's _backward in reverse topological order — guaranteeing each node's grad is fully accumulated before it propagates further back. That ordering is the only 'algorithm' in backpropagation; everything else is the local derivatives from Day 22.
def backward(self):
topo, visited = [], set()
def build(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build(child)
topo.append(v)
build(self)
self.grad = 1.0 # dOutput/dOutput = 1
for node in reversed(topo): # process outputs before their inputs
node._backward()A neuron, then a layer, then an MLP
A neuron computes activation(w·x + b) — a weighted sum of inputs plus a bias, passed through a nonlinearity like tanh or ReLU. Stack neurons into a layer, stack layers into a multi-layer perceptron (MLP). Because every operation is built from micrograd Values, calling .backward() on the final loss populates the gradient of *every* weight automatically. You've just built a trainable neural network from scalar arithmetic.
This is genuinely what PyTorch does
PyTorch replaces micrograd's scalar Value with tensor operations on the GPU and adds thousands of ops, but the architecture is identical: record a graph on the forward pass, walk it backward in topological order applying local derivatives. You will never again wonder what .backward() does.
Key terms
- Backpropagation
- Computing gradients by traversing the computation graph in reverse topological order, applying the chain rule at each node.
- Neuron
- A unit computing a nonlinear function of a weighted sum of its inputs plus a bias: activation(w·x + b).
- Multi-layer perceptron (MLP)
- A neural network of stacked fully-connected layers with nonlinear activations between them.
Why must backpropagation visit graph nodes in reverse topological order?
