← AI Terminology

Activation Function

An activation function is a mathematical function applied to a neuron's output that introduces non-linearity, allowing neural networks to learn complex patterns beyond simple linear relationships.

Without activation functions, stacking layers would collapse to a single linear transformation no matter how deep the network.
Why It Matters in AI
Activation functions are what give neural networks their expressive power. A network of purely linear layers can only represent linear functions; a single ReLU layer breaks that ceiling. The choice of activation function affects training speed, gradient flow, and whether a network can even learn at all (vanishing gradient problem).
Key Points
Aspect Description
GELU Smooth ReLU variant used in Transformers (GPT, BERT); slightly outperforms ReLU on NLP tasks
ReLU max(0, x) — default for hidden layers; fast, sparse, but can "die" (zero gradient for x < 0)
Tanh Maps to (−1, 1); older default; still used in LSTMs and some RNNs
SwiGLU Gated variant used in Llama/PaLM; currently dominant in frontier LLMs
Sigmoid Maps output to (0,1); used in binary output layers; suffers vanishing gradients in deep networks
Softmax Converts a vector to a probability distribution; standard for multi-class classification output
Simple Analogy
Think of a light dimmer vs an on/off switch. A linear neuron is binary — on or off. An activation function is the dimmer: it lets the neuron respond to inputs with varying intensity and shape, allowing the network to model smooth, complex curves.
Common Usage Examples
  • nn.ReLU() — the standard hidden-layer activation in PyTorch
  • nn.GELU() — used in HuggingFace Transformer models
  • torch.nn.functional.softmax(logits, dim=-1) — final layer of a classifier
  • Dying ReLU problem: neurons stuck at 0 — fixed by Leaky ReLU (nn.LeakyReLU)
  • SwiGLU in Meta's Llama models replacing ReLU for better throughput and accuracy
Summary
In short: Activation functions are the non-linearity that turns a stack of matrix multiplications into a universal function approximator.