← AI Terminology
Residual Connection
A residual connection (skip connection) is a shortcut path in a neural network that adds the input of a layer directly to its output — f(x) + x — allowing gradients to flow directly back through the shortcut without passing through the layer's transformations, enabling training of very deep networks.
Introduced in ResNet (He et al., 2015); now universal in all deep learning architectures.
Introduced in ResNet (He et al., 2015); now universal in all deep learning architectures.
Why It Matters in AI
Without skip connections, training networks deeper than ~20 layers leads to degradation: the accuracy saturates and then declines — not from overfitting but from optimisation difficulty. Residual connections solve this: if a layer is not useful, the network can learn the identity mapping (weights → 0) and the skip path carries the signal. This allows training of networks with hundreds of layers (ResNet-1001) and is why transformers can be stacked arbitrarily deep. Every modern deep learning architecture — ResNet, BERT, GPT, ViT, U-Net — uses residual connections.
Key Points
| Aspect | Description |
|---|---|
| Formula | y = F(x, W) + x — layer output added to layer input before next block |
| Transformers | x = x + Attention(LayerNorm(x)); x = x + FFN(LayerNorm(x)) — residuals in every transformer block |
| Pre-activation | He et al. 2016: BatchNorm → ReLU → Conv (before residual add) — better than post-activation |
| Gradient highway | Gradient flows directly through + without passing through F — prevents vanishing gradients |
| Identity mapping | If F(x) → 0, the block becomes identity: y = x — graceful degradation as depth increases |
| Dense connections | DenseNet: skip connections from every layer to every subsequent layer — maximum gradient flow |
Simple Analogy
An express train track running parallel to a local track: the local train (layer transformations) makes stops and can slow down; the express track (skip connection) runs directly from origin to destination. If the local track is congested, signal continues on the express track — the network never loses the original input completely.
Common Usage Examples
- ResNet block:
output = F.relu(bn(conv(x)) + x)— standard residual block - Pre-activation:
output = conv(relu(bn(x))) + x— improved variant - Transformer encoder:
x = x + self.attn(self.norm1(x)); x = x + self.ffn(self.norm2(x)) torchvision.models.resnet50()— all 50 layers connected via residual shortcuts- U-Net: encoder skip connections to decoder — preserves spatial detail across resolution changes
Summary
In short: Residual connections add the layer input directly to its output — creating gradient highways that prevent vanishing gradients and enable training of arbitrarily deep networks, making them a universal component of every modern architecture.