← AI Terminology
SwiGLU - Swish-Gated Linear Unit
SwiGLU is a gated feed-forward activation: SwiGLU(x) = Swish(xW) ⊗ (xV), combining a SiLU/Swish gate with a linear path (plus a down-projection).
Popularised by PaLM-era work and now standard in LLaMA, Mistral, Gemma, and many open LLMs.
Popularised by PaLM-era work and now standard in LLaMA, Mistral, Gemma, and many open LLMs.
Why It Matters in AI
Feed-forward layers dominate transformer parameter count; swapping a plain GELU-MLP for SwiGLU consistently improves quality at similar compute. Understanding SwiGLU is essential to reading modern LLM architecture cards.
Key Points
| Aspect | Description |
|---|---|
| Gate | SiLU/Swish: x · σ(x) |
| Paper | GLU Variants Improve Transformer (Shazeer, 2020) |
| Related | GeGLU, ReGLU — other gated linear unit variants |
| Used in | LLaMA, Mistral, Mixtral, Gemma, PaLM-style models |
| Structure | Two up-projections → elementwise product → down-projection |
| Width note | Often ~2/3 of GELU-FFN width to match param count |
Simple Analogy
Instead of one kitchen pass, ingredients go through a seasoning gate and a main path, then multiply — only flavours the gate opens get through strongly.
Common Usage Examples
- LLaMA MLP:
down_proj(silu(gate_proj(x)) * up_proj(x)) - Hugging Face
LlamaMLP - Config: gated FFN with
silu - Ablation: GELU-FFN vs SwiGLU at fixed FLOPs
Summary
In short: SwiGLU is the gated MLP activation used by most modern open LLMs — a small structural change that reliably boosts language-model quality.