← AI Terminology
Backpropagation
Backpropagation is the algorithm that computes gradients of the loss function with respect to every weight in a neural network, enabling those weights to be updated via gradient descent.
It works by applying the chain rule of calculus backwards from the output layer to the input layer.
It works by applying the chain rule of calculus backwards from the output layer to the input layer.
Why It Matters in AI
Backpropagation is what makes training deep networks feasible. Without it, computing gradients for millions of parameters would be intractable. Its invention (popularised by Rumelhart, Hinton & Williams in 1986) is widely credited as the breakthrough that made modern deep learning possible — every neural network trained today uses backpropagation or a generalisation of it.
Key Points
| Aspect | Description |
|---|---|
| Chain rule | ∂L/∂w = ∂L/∂a × ∂a/∂w — gradients flow back through composite functions layer by layer |
| Complexity | O(number of parameters) — same order as the forward pass, making it efficient |
| Forward pass | Compute predictions and loss — saves intermediate activations for the backward pass |
| Backward pass | Propagate loss gradient back through layers using chain rule — computes ∂Loss/∂w for every w |
| Automatic diff | Modern frameworks (PyTorch autograd, JAX, TensorFlow GradientTape) do backprop automatically |
| Vanishing grad | Deep networks: gradients shrink toward zero through many layers — fixed by ReLU, ResNets, LN |
Simple Analogy
Baking a cake and tasting it — the output tells you it's too salty. Backpropagation is the process of tracing backwards from "too salty" through each ingredient's contribution to determine how much to reduce the salt. Each step back through the recipe assigns credit (or blame) proportionally.
Common Usage Examples
loss.backward()in PyTorch — triggers automatic backpropagation through the computation graphtape.gradient(loss, model.trainable_variables)in TensorFlow GradientTape- Gradient clipping applied after
loss.backward()to prevent exploding gradients - Vanishing gradient in early RNNs — reason LSTMs and GRUs were invented
torch.autograd.grad()for computing higher-order gradients (used in MAML meta-learning)
Summary
In short: Backpropagation is the algorithm that tells every weight in a neural network exactly how to change to reduce the error — without it, deep learning as we know it would not exist.