← AI Terminology
MoE - Mixture of Experts
Mixture of Experts is a neural network architecture where a model contains multiple specialised sub-networks ("experts") and a routing mechanism that selects which experts process each input token — activating only a subset of parameters per token, enabling massive parameter counts without proportional compute cost.
GPT-4, Mixtral 8x7B, Grok-1, and Gemini 1.5 all use MoE architectures.
GPT-4, Mixtral 8x7B, Grok-1, and Gemini 1.5 all use MoE architectures.
Why It Matters in AI
Scaling laws say more parameters = better models, but dense models require proportional compute for every token. MoE decouples parameter count from FLOPs per token: a model with 560B total parameters but only 56B active per token (10% sparsity) achieves near-dense-560B quality at near-dense-56B inference cost. This is how frontier models achieve both extreme scale and practical inference costs — MoE is the architecture enabling the next generation of LLMs.
Key Points
| Aspect | Description |
|---|---|
| Router | A learned gating network: for each token, outputs a probability distribution over all experts |
| Sparse MoE | Only k of N experts activate per token — the standard MoE variant in LLMs |
| Communication | Multi-GPU: experts distributed across GPUs, requiring all-to-all communication — challenging |
| Expert layers | Typically replace FFN sublayers in transformer blocks — attention layers remain dense |
| Top-k routing | Select the top-k experts per token (typically k=1 or k=2) — only they process the token |
| Load balancing | Auxiliary loss penalises uneven expert utilisation — prevents all tokens routing to one expert |
Simple Analogy
A law firm with specialists: instead of every partner (dense model) reading every brief, a routing coordinator assigns each brief to the two most relevant specialists. The firm handles far more cases than any generalist could — but only two experts read each brief, keeping per-case cost low.
Common Usage Examples
- Mixtral 8x7B: 8 experts per MoE layer, top-2 routing — 47B total params, ~13B active per token
from transformers import MixtralForCausalLM— HuggingFace Mixtral MoE implementationtorch.nn.Linearper expert, gating:gates = softmax(x @ W_gate)[:, :top_k]- Auxiliary load balancing loss:
aux_loss = cv_squared(expert_counts)— penalises routing imbalance - Google Switch Transformer: showed MoE with top-1 routing scales efficiently to 1.6T parameters
Summary
In short: Mixture of Experts enables massive parameter counts at manageable compute cost by routing each token to only a small subset of specialist networks — the key architectural innovation behind the largest frontier LLMs.