← AI Terminology
Mixture of Depths
Mixture of Depths (MoD) routes tokens to use more or fewer layers/compute dynamically, so easy tokens take cheap paths and hard tokens take deeper computation.
It is conditional computation along depth rather than experts (MoE).
It is conditional computation along depth rather than experts (MoE).
Why It Matters in AI
Not all tokens need full stack compute. MoD-style routing saves FLOPs while preserving quality, complementing MoE’s expert routing. Efficiency research increasingly explores dynamic depth.
Key Points
| Aspect | Description |
|---|---|
| Idea | Learn which tokens skip or exit early per block |
| Status | Active research; selective production interest |
| Vs MoE | MoE routes across experts; MoD across depth/compute |
| Benefit | Lower average FLOPs per token |
| Related | Early exit, MoE, adaptive computation |
| Challenge | Routing stability and batching efficiency |
Simple Analogy
A school where advanced students skip basic modules while others take full depth — compute matches difficulty.
Common Usage Examples
- MoD papers on dynamic token routing
- Compare FLOPs vs quality to dense baselines
- Batch implementations of mixed-depth tokens
- Pair with MoE for dual conditional compute
Summary
In short: Mixture of Depths spends more layers on hard tokens and fewer on easy ones — adaptive depth for efficient transformers.