← AI Terminology
ReLU - Rectified Linear Unit
ReLU (Rectified Linear Unit) is an activation function defined as f(x) = max(0, x) — outputting the input if positive, zero otherwise — that introduced non-linearity to deep neural networks with minimal compute cost and dramatically faster training than sigmoid/tanh.
It became the default activation function in deep learning from 2012 onward.
It became the default activation function in deep learning from 2012 onward.
Why It Matters in AI
Before ReLU, networks used sigmoid and tanh activations — both suffer from vanishing gradients for large inputs (output saturates near 0 or 1, gradient → 0). ReLU has a constant gradient of 1 for positive inputs — gradients flow freely through positive activations. This made training deep networks practical for the first time: AlexNet (2012) used ReLU and outperformed prior work dramatically. Modern variants (GELU, SiLU, Swish) outperform ReLU in transformers, but ReLU remains standard in CNNs.
Key Points
| Aspect | Description |
|---|---|
| ELU | Exponential Linear Unit — smooth for x < 0; better than Leaky ReLU for some applications |
| GELU | Gaussian Error Linear Unit — smooth approximation of ReLU; standard in transformers (BERT, GPT) |
| Formula | f(x) = max(0, x) — simplest piecewise linear function; gradient is 0 or 1 |
| Dying ReLU | Neurons with negative pre-activations always output 0 — they "die" and stop learning |
| Leaky ReLU | f(x) = max(0.01x, x) — small negative slope prevents dying neurons |
| SiLU / Swish | f(x) = x × sigmoid(x) — smooth, bounded-below nonlinearity; used in Llama, MobileNetV3 |
Simple Analogy
A one-way valve: positive signals pass through unchanged; negative signals are blocked (zero). This simple valve lets useful forward signals flow while suppressing irrelevant ones — and crucially, the valve doesn't create new bottlenecks (no gradient vanishing for positive signals).
Common Usage Examples
nn.ReLU()orF.relu(x)— standard PyTorch ReLU layer/function- CNN architecture:
Conv2d → BatchNorm → ReLU— canonical convolutional block nn.GELU()orF.gelu(x)— transformer FFN activation (BERT, GPT-2, GPT-3)nn.SiLU()— SiLU/Swish activation used in Llama 2/3 FFN sublayers- Leaky ReLU in GANs:
nn.LeakyReLU(0.2)— prevents dying ReLU in discriminator networks
Summary
In short: ReLU is the activation function that enabled deep learning — its constant positive gradient eliminates vanishing gradients, and its computational simplicity made training large CNNs practical, remaining the default for convolutional networks while GELU/SiLU dominate transformers.