← AI Terminology
Vocabulary
A vocabulary (or tokenizer vocabulary) is the fixed set of tokens a language model can process — the complete list of subword units, words, or characters mapped to integer IDs that the model uses as its input and output alphabet.
Vocabulary size is a fundamental hyperparameter that trades coverage against embedding table size.
Vocabulary size is a fundamental hyperparameter that trades coverage against embedding table size.
Why It Matters in AI
A model can only produce tokens in its vocabulary — anything outside it is either broken into subword pieces or handled by a special
<unk> token. Vocabulary size directly controls the embedding matrix size (vocab_size × embedding_dim), which can be hundreds of MB. Too small a vocabulary means rare words fragment into many tokens (slow, lossy); too large inflates the embedding table and softmax computation. GPT-4 uses ~100K tokens; Llama 3 expanded from 32K to 128K to improve non-English and code coverage.Key Points
| Aspect | Description |
|---|---|
| Size | Typical ranges: 32K (Llama 2), 50K (GPT-2), 100K (GPT-4/Llama 3), 250K (Gemma) |
| Coverage | Larger vocabulary = fewer tokens per sentence = faster inference; better multilingual coverage |
| Construction | Learned via BPE, WordPiece, or SentencePiece on a large corpus — reflects training data distribution |
| Tokenisation | Text → sequence of token IDs by looking up the vocabulary; reverse is detokenisation |
| Special tokens | <bos>, <eos>, <pad>, <unk> — sentinel tokens for start/end/padding/unknown |
| Embedding table | nn.Embedding(vocab_size, d_model) — the vocabulary's learnable representation matrix |
Simple Analogy
A typewriter's character set: you can only type characters that exist on the keys. A typewriter with 30 keys can write English but not Chinese; one with 250 keys covers most scripts. A model's vocabulary is its typewriter — everything it can "say" must be assembled from its fixed set of keys.
Common Usage Examples
tokenizer.vocab_size— check vocabulary size of any HuggingFace tokenizertokenizer.convert_tokens_to_ids(["hello", "world"])— lookup token IDstokenizer.get_vocab()— returns the full{token: id}dictionary- Adding tokens:
tokenizer.add_tokens(["<tool_call>"]); model.resize_token_embeddings(len(tokenizer)) - OOV handling:
tokenizer("naïve", add_special_tokens=False)— splits into subword pieces
Summary
In short: A model's vocabulary is its fixed alphabet of tokens — the complete set of subword units it can read and write, whose size governs the tradeoff between multilingual coverage and memory cost.