← AI Terminology
Dropout
Dropout is a regularisation technique where, during each training step, a random fraction of neurons is temporarily disabled (set to zero) — preventing neurons from co-adapting too closely and forcing the network to learn more robust, distributed representations.
Introduced by Srivastava et al. (2014) at Toronto, it was one of the most impactful regularisation methods in deep learning.
Introduced by Srivastava et al. (2014) at Toronto, it was one of the most impactful regularisation methods in deep learning.
Why It Matters in AI
Deep networks with millions of parameters easily overfit to training data. Dropout acts as a cheap ensemble method: by randomly dropping neurons, each training step effectively trains a different sub-network, and inference averages over an exponential number of these sub-networks. It dramatically reduces overfitting without increasing computational cost significantly.
Key Points
| Aspect | Description |
|---|---|
| Where used | Fully connected layers, Transformer attention/FFN layers; rarely in CNN conv layers |
| Alternatives | DropBlock (spatial dropout for CNNs), DropPath (for ViT), Weight Decay (often more effective for Transformers) |
| Training only | Neurons dropped during training; all neurons active at inference (outputs scaled by 1/(1−p)) |
| Interpretation | Approximate ensemble of 2^n sub-networks — each training step trains a different sub-network |
| Drop probability | Typical values: 0.1–0.5 for hidden layers; 0.1 for Transformers; 0.5 for older FC networks |
| Inverted dropout | Scale surviving activations by 1/(1−p) during training — simpler, more common implementation |
Simple Analogy
Practising a speech where random team members are absent each day — you can never rely on any single person, so everyone must learn all parts. The final performance (inference with everyone present) benefits from this distributed preparation. No single neuron becomes indispensable.
Common Usage Examples
nn.Dropout(p=0.1)in a Transformer FFN block in PyTorchmodel.train()activates dropout;model.eval()disables it — always calleval()at inference- MC Dropout: keep dropout active at inference time, run 100 forward passes — estimate uncertainty
- HuggingFace
BertConfig(hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1) - DropPath (stochastic depth): randomly drop entire residual branches in deep ViTs during training
Summary
In short: Dropout randomly silences neurons during training — forcing the network to learn redundant, distributed representations rather than memorising the training set.