← AI Terminology
Sparse MoE - Sparse Mixture of Experts
Sparse Mixture of Experts is the standard MoE architecture used in LLMs where only a small number k of N expert networks (typically k=2 of N=8 or N=64) are activated per token — making compute per token proportional to k, not N, enabling massive parameter counts with manageable inference cost.
"Sparse" contrasts with dense MoE (all experts active) — sparse is the practical production variant.
"Sparse" contrasts with dense MoE (all experts active) — sparse is the practical production variant.
Why It Matters in AI
Dense MoE activates all experts for every token — same compute as a single large model. Sparse MoE activates only top-k experts — typically 2 of 8 — keeping FLOPs at 1/4 the total-parameter cost. This is why Mixtral 8x7B (47B total parameters) costs similar compute to a 13B dense model per token, while achieving quality closer to 70B. Sparse MoE is the architecture enabling frontier models to have enormous parameter capacity (intelligence) at practical serving cost.
Key Points
| Aspect | Description |
|---|---|
| Memory | Full parameter set in GPU HBM — all expert weights loaded even if only k active per token |
| Router | Learned gating: scores = softmax(x @ W_g); top_k_experts = scores.topk(k) |
| Sparsity | k of N experts active: Mixtral k=2 of N=8; Grok-1 k=2 of N=8; Switch k=1 of N=2048 |
| Token choice | Each token picks its top-k experts; Expert choice: each expert picks its top-T tokens (no drop) |
| Load balancing | Auxiliary loss prevents all tokens routing to same experts — expert capacity limits enforced |
| Expert capacity | Maximum tokens per expert per batch — overflow tokens are "dropped" or passed through |
Simple Analogy
A hospital triage system: each patient (token) is routed to the 2 most relevant specialists (experts) for their condition — not all 8 specialists in the hospital. The hospital (model) has 8 specialists' worth of knowledge, but the cost per patient is 2 specialists' time. Sparse MoE is this specialised routing applied to neural network computation.
Common Usage Examples
- Mixtral 8x7B:
AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1")— sparse MoE LLM from megablocks import dMoE; moe_layer = dMoE(model_dim=4096, num_experts=8, top_k=2)— efficient MoE- Expert capacity:
capacity_factor=1.25— each expert handles up to 1.25× the average tokens per batch - Load balancing:
aux_loss = load_balancing_loss_func(router_logits, top_k=2)— HuggingFace implementation - Mixtral inference:
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1— vLLM handles sparse MoE natively
Summary
In short: Sparse MoE activates only k of N expert networks per token — enabling models with 8× the parameters of a dense model at ~2× the compute cost, the architectural innovation behind Mixtral, Grok-1, and GPT-4's efficiency.