← AI Terminology
Context Window
A context window is the maximum amount of text (measured in tokens) that a language model can process in a single forward pass — both the input it reads and the output it generates must fit within this limit.
It defines the model's "working memory": anything outside the window is invisible to the model.
It defines the model's "working memory": anything outside the window is invisible to the model.
Why It Matters in AI
The context window determines what tasks an LLM can perform in one pass: summarise a 10-page document, analyse a full codebase, hold a long conversation. Short windows force chunking and retrieval workarounds; long windows enable end-to-end reasoning on large inputs. The race to extend context windows from 4K (GPT-3) to 1M+ tokens (Gemini 1.5) is one of the defining capability improvements of recent LLMs.
Key Points
| Aspect | Description |
|---|---|
| KV cache | Stores key-value pairs for all context tokens — scales linearly in memory with window size |
| Techniques | RoPE, ALiBi (positional bias), Flash Attention, sliding window attention extend effective length |
| Measured in | Tokens — ~4 chars/token in English; a 100K token window ≈ ~75K words ≈ a full novel |
| Modern sizes | GPT-4o: 128K; Claude 3.5 Sonnet: 200K; Gemini 1.5 Pro: 1M; Llama 3.1: 128K |
| Attention cost | Self-attention is O(n²) in sequence length — longer windows require architectural optimisations |
| Long-context perf | "Lost in the middle": models attend less to content in the middle of very long contexts |
Simple Analogy
A context window is the model's desk space. Everything on the desk it can see and reason about simultaneously; everything else is in filing cabinets (inaccessible without retrieval). A bigger desk lets you spread out a full case file; a small desk forces you to shuffle papers in and out.
Common Usage Examples
- Claude 3.5 Sonnet's 200K context: analyse an entire codebase or legal contract in one call
- Gemini 1.5 Pro's 1M context: process an entire feature-length film's worth of video frames
- RAG (Retrieval-Augmented Generation): retrieves relevant chunks when content exceeds the window
tiktoken.encode(text)— count tokens before sending to OpenAI API to avoid context limit errors- Long-context eval: "needle in a haystack" test — can the model find a planted fact in a 100K-token document?
Summary
In short: The context window is how much text the model can see at once — larger windows unlock longer documents and richer reasoning, but require more memory and careful architectural engineering.