Day 21: Derivatives & the chain rule, by hand
The one piece of calculus that runs all of deep learning
Everything from here to Stage 6 rests on one idea: a derivative measures how much a function's output changes when you nudge its input. If you know that nudging a parameter *up* makes the error go *down*, you know which way to move it. Neural network training is nothing more than: compute the error, find each parameter's derivative of that error, nudge every parameter slightly in the direction that reduces it, repeat. Karpathy's *Zero-to-Hero* series (which you start tomorrow) makes this concrete in code — today builds the intuition.
Derivative as slope
For f(x) = x², the derivative is 2x. At x=3 the slope is 6: nudge x up by a tiny ε and f goes up by about 6ε. A *positive* derivative means increasing x increases f; to *decrease* f you move x in the opposite direction of the derivative. That last sentence is the whole of gradient descent.
def f(x):
return x**2
def numerical_derivative(f, x, eps=1e-6):
return (f(x + eps) - f(x)) / eps
numerical_derivative(f, 3.0) # ~6.0, matches 2x at x=3The chain rule
Neural networks are functions of functions of functions — deeply nested. The chain rule says the derivative of a composition is the *product* of the derivatives along the chain: if y = f(g(x)), then dy/dx = f'(g(x)) · g'(x). Backpropagation (tomorrow) is literally the chain rule applied mechanically backwards through a network, multiplying local slopes to find how each early parameter affects the final error.
Key terms
- Derivative
- The instantaneous rate of change of a function's output with respect to a small change in its input; the slope.
- Gradient
- The vector of partial derivatives of a function with respect to all its inputs/parameters at once.
- Chain rule
- The rule for differentiating composed functions: multiply the derivatives of each link in the chain.
A parameter w has a positive derivative of the loss (dLoss/dw > 0). To reduce the loss, which way should you nudge w?
