← AI Terminology
Tokenization
Tokenization is the process of splitting raw text into a sequence of tokens — the discrete units that a language model processes — using an algorithm (BPE, WordPiece, SentencePiece) that builds a vocabulary by merging frequent character pairs, balancing vocabulary size against sequence length.
Every LLM has its own tokenizer with a fixed vocabulary that must be used consistently at training and inference.
Every LLM has its own tokenizer with a fixed vocabulary that must be used consistently at training and inference.
Why It Matters in AI
Tokenization has significant practical consequences: "tokenizer mismatch" (using the wrong tokenizer) produces wrong token IDs and broken inference. Tokenization determines how efficiently text is compressed into tokens — English is compact (~1.3 tok/word); some languages (Thai, Arabic) are much less efficient, increasing API costs and reducing effective context window capacity for non-English text. Poor tokenization is the source of many LLM "failures" with arithmetic (numbers are tokenized inconsistently) and spelling (tokens span characters in unexpected ways).
Key Points
| Aspect | Description |
|---|---|
| BPE | Byte Pair Encoding: start with characters, iteratively merge most frequent adjacent pairs — GPT tokenizers |
| WordPiece | Like BPE but maximises likelihood; used by BERT — produces ##word sub-word markers |
| Byte fallback | Modern tokenizers can represent any Unicode character via byte-level fallback — no unknowns |
| SentencePiece | Language-agnostic, trains on raw bytes — used by LLaMA, T5, Gemma |
| Special tokens | <bos>, <eos>, <pad>, [SEP], [MASK] — model-specific structural tokens |
| Vocabulary size | GPT-2: 50,257 tokens; Llama 3: 128,256 tokens — larger vocab = fewer tokens per text |
Simple Analogy
A shorthand notation system: a stenographer doesn't write every letter — they use symbols for common words and sounds. Tokenization creates a similar shorthand: common words get a single token ID; rare words are broken into sub-word pieces. The goal is to represent any text with the minimum number of tokens from a fixed vocabulary.
Common Usage Examples
from transformers import AutoTokenizer; tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")tokens = tokenizer.encode("Hello, world!"); tokenizer.decode(tokens)— encode → decode roundtriptokenizer.batch_encode_plus(texts, padding=True, truncation=True, max_length=512)— batched tokenization- tiktoken:
enc = tiktoken.get_encoding("cl100k_base"); enc.encode("OpenAI uses this for GPT-4") - SentencePiece:
import sentencepiece as spm; sp.load("llama.model"); sp.encode("Hello")
Summary
In short: Tokenization splits raw text into tokens using learned subword vocabularies — the critical preprocessing step that determines how efficiently any text is represented, affecting context window capacity, API costs, and model behaviour on edge cases like numbers and spelling.