← AI Terminology
Weight Tying
Weight tying shares the input token embedding matrix with the output language-model head (or a transpose), reducing parameters and often helping perplexity.
Common in GPT-2-style and many compact language models.
Common in GPT-2-style and many compact language models.
Why It Matters in AI
Embedding tables are huge at large vocabularies; tying them to the softmax head halves that cost and links 'read token' and 'predict token' spaces. Standard when reading parameter counts and export formats.
Key Points
| Aspect | Description |
|---|---|
| Benefit | Fewer parameters; shared input/output geometry |
| HF flag | tie_word_embeddings=True |
| Related | Embedding factorisation for huge vocabs |
| Tradeoff | Less flexible than untied heads (some large models untie) |
| Popularised | Press & Wolf; Inan et al.; GPT-2 practice |
| What is shared | token embedding weights and lm_head weights |
Simple Analogy
Using the same bilingual dictionary both to look up words when reading and to choose words when writing — one lexicon, two jobs.
Common Usage Examples
model.lm_head.weight = model.embed_tokens.weight- GPT-2 tied embeddings
- Check
config.tie_word_embeddings - Avoid duplicating tied weights when exporting
Summary
In short: Weight tying reuses the input embedding matrix as the output projection — saving parameters and binding input and output token spaces.