← AI Terminology
Sigmoid
Sigmoid is an activation function σ(x) = 1/(1+e^(−x)) that maps any real number to the open interval (0, 1).
It is the classic binary-classification output and a building block of gating mechanisms in LSTMs, GRUs, and some GLU layers.
It is the classic binary-classification output and a building block of gating mechanisms in LSTMs, GRUs, and some GLU layers.
Why It Matters in AI
Binary decisions need a calibrated probability; sigmoid provides that. Gates that open or close information flow also rely on its (0,1) range. Though ReLU/GELU dominate hidden layers, sigmoid remains essential at binary heads and inside recurrent/gated units.
Key Points
| Aspect | Description |
|---|---|
| Gates | Forget/input/output gates in LSTM use sigmoid |
| Range | (0, 1) — saturates near extremes |
| Binary CE | Paired with binary cross-entropy for two-class problems |
| Derivative | σ'(x) = σ(x)(1−σ(x)) — small when saturated (vanishing gradients) |
| Modern use | Standard for multi-label and binary heads; rare as hidden activation |
| Vs softmax | Sigmoid is independent per logit; softmax is mutually exclusive multi-class |
Simple Analogy
A dimmer switch from fully off toward fully on: mid values are possible, but push hard enough and it clamps near 0 or 1.
Common Usage Examples
torch.sigmoid(x)orx.sigmoid()nn.BCEWithLogitsLoss— stable sigmoid + BCE- Multi-label: independent sigmoid per class
- LSTM:
f_t = σ(W_f · [h, x])
Summary
In short: Sigmoid squashes any score into a (0,1) probability — the classic tool for binary outputs and soft gates.