← AI Terminology
Token
A token is the basic unit of text that a language model processes — a fragment of text produced by a tokenizer that may be a word, part of a word, a punctuation mark, or a special symbol — with each language model operating on sequences of tokens rather than raw characters or words.
"1 token ≈ 0.75 words" for English text with GPT-style tokenizers.
"1 token ≈ 0.75 words" for English text with GPT-style tokenizers.
Why It Matters in AI
Everything in an LLM — context window, pricing, throughput, attention complexity — is measured in tokens. A model's "128K context window" means it can process 128,000 tokens simultaneously. API pricing is always per-million-tokens-in/out. The tokenizer determines how text maps to tokens: "unhappiness" might be 1 token ("unhappiness") or 3 ("un", "happi", "ness") — affecting cost and context utilisation. Understanding tokens is fundamental to working with LLM APIs efficiently.
Key Points
| Aspect | Description |
|---|---|
| BPE | Byte Pair Encoding — most common tokenization algorithm; merges frequent character pairs |
| Pricing | OpenAI/Anthropic APIs: charged per input token + per output token separately |
| Vocabulary | A fixed set of tokens (typically 32K–200K) — each token has a unique integer ID |
| Token density | English: ~1.3 tokens/word; code: denser; Chinese/Japanese: ~1 char = ~1–2 tokens |
| Context window | Maximum token count a model can process at once — input + output combined for most models |
| Special tokens | `< |
Simple Analogy
LEGO bricks for language: instead of working letter-by-letter (too granular) or word-by-word (too limited a vocabulary), LLMs work with pre-defined bricks of varying sizes. Common words are one brick; rare words are assembled from several smaller bricks. The model's skill is assembling meaningful structures from its fixed brick set.
Common Usage Examples
from tiktoken import encoding_for_model; enc = encoding_for_model("gpt-4"); enc.encode("Hello world")→[9906, 1917]len(enc.encode(text))— count tokens before sending to API to estimate cost- HuggingFace:
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B"); tokenizer.encode("Hello") - Pricing: GPT-4o input $2.50/1M tokens; output $10/1M tokens — multiply by token count for cost estimate
- Context utilisation: 128K context window − 1K system prompt − 50K docs = 77K tokens for conversation
Summary
In short: A token is the atomic unit of text in language models — roughly 0.75 of a word in English — determining context window capacity, API pricing, throughput, and every other operational metric of LLM systems.