← AI Terminology
BPE - Byte Pair Encoding
BPE stands for Byte Pair Encoding: a subword tokenization algorithm that starts with individual characters and iteratively merges the most frequent adjacent pair of symbols until reaching a target vocabulary size.
It is the tokenization method used by GPT-2, GPT-4, Llama, and most modern LLMs.
It is the tokenization method used by GPT-2, GPT-4, Llama, and most modern LLMs.
Why It Matters in AI
LLMs need a fixed vocabulary to operate, but any fixed word-level vocabulary leaves rare words unknown. BPE solves this: common words become single tokens ("the", "Paris") while rare words decompose into subwords ("uncommon" → "un", "common"). This gives a compact vocabulary that handles any text without unknown tokens. The tokenizer determines what text looks like to the model — BPE shapes how LLMs represent and process language.
Key Points
| Aspect | Description |
|---|---|
| Algorithm | Count all adjacent symbol pairs, merge the most frequent, repeat until vocab size reached |
| Alternatives | WordPiece (BERT), SentencePiece (T5, Llama), Unigram LM — all achieve similar goals |
| Byte-level BPE | GPT-2 uses bytes as the base alphabet — handles any Unicode character without unknowns |
| Vocabulary size | GPT-4: ~100K tokens; Llama: 32K; BERT (WordPiece variant): 30K |
| Impact on models | Token count affects context window usage — short languages can express more in fewer tokens |
| Subword handling | "tokenization" → ["token", "ization"] — rare compound split, common parts preserved as tokens |
Simple Analogy
Imagine building a dictionary by starting with all individual letters, then repeatedly finding the two symbols that appear most often side-by-side and combining them into a new symbol. "th" → single symbol; "the" → single symbol; eventually you have a vocabulary of the most useful chunks, not arbitrary word boundaries.
Common Usage Examples
tiktoken.encoding_for_model("gpt-4").encode("Hello, world!")— GPT-4's BPE tokenizer- HuggingFace
tokenizerslibrary:ByteLevelBPETokenizertrains a custom BPE vocab on any corpus from transformers import AutoTokenizer; tokenizer = AutoTokenizer.from_pretrained("gpt2")- Token count matters for billing: OpenAI charges per token — roughly 4 chars = 1 token in English
- Coding models: code has different frequency patterns, so code-specific BPE vocabularies are trained
Summary
In short: BPE is the tokenization algorithm that breaks text into the most useful sub-word chunks — giving LLMs a compact vocabulary that handles any language without unknown words.