← AI Terminology
GELU - Gaussian Error Linear Unit
GELU (Gaussian Error Linear Unit) is a smooth activation: x · Φ(x), where Φ is the standard normal CDF — roughly a soft ReLU weighted by a Gaussian.
It is the default hidden activation in BERT, GPT-2/3-era transformers, and many models that have not switched to SwiGLU.
It is the default hidden activation in BERT, GPT-2/3-era transformers, and many models that have not switched to SwiGLU.
Why It Matters in AI
ReLU zeros out negatives hard; GELU’s smooth form often trains transformers more stably and accurately. Choosing GELU (or SwiGLU) over ReLU is a small architectural detail with large empirical impact on language models.
Key Points
| Aspect | Description |
|---|---|
| Origin | Hendrycks & Gimpel (2016) |
| Formula | GELU(x) ≈ 0.5x(1+tanh[√(2/π)(x+0.044715x³)]) |
| PyTorch | F.gelu / nn.GELU() |
| Used in | BERT, GPT-2, ViT, many Hugging Face defaults |
| Vs ReLU | Smoother; small negative pass-through near zero |
| Vs SwiGLU | SwiGLU often wins in modern LLMs; GELU still ubiquitous |
Simple Analogy
ReLU is a hard gate: negative means off. GELU is a soft probabilistic gate — values near zero are only partially let through.
Common Usage Examples
nn.GELU()in Transformer MLP blocks- BERT: GELU throughout feed-forward layers
F.gelu(x, approximate='tanh')for speed- HF config:
hidden_act: gelu
Summary
In short: GELU is the smooth, Gaussian-weighted activation that replaced ReLU as the default non-linearity inside many transformer MLPs.