← AI Terminology
In-Context Learning
In-context learning (ICL) is the ability of large language models to perform new tasks from a handful of examples provided in the prompt — without any gradient updates to the model's weights.
It was a surprising emergent capability demonstrated at scale by GPT-3 (Brown et al., 2020).
It was a surprising emergent capability demonstrated at scale by GPT-3 (Brown et al., 2020).
Why It Matters in AI
ICL changes how AI is deployed: instead of training a task-specific model (expensive, slow), you provide examples in the prompt and the model adapts immediately. This makes LLMs general-purpose tools that can be redirected to new tasks in seconds. ICL is why GPT-4 can learn to format JSON, adopt a persona, or solve a new puzzle type — all from a few examples in the conversation.
Key Points
| Aspect | Description |
|---|---|
| Few-shot | 1–5 examples in the prompt — model infers the input-output pattern |
| Many-shot | 10s–100s of examples (enabled by long context windows) — approaches fine-tuning performance |
| Mechanism | Debated: implicit gradient descent in attention heads? Bayesian inference? Active research |
| Zero-shot | No examples — the task is described in natural language only |
| No gradient update | Weights are frozen — the model learns from context statistically, not by parameter update |
| Example sensitivity | Order and format of examples affect performance — not always robust |
Simple Analogy
Showing a new employee three email examples labelled "URGENT" and three labelled "NOT URGENT" — they can now classify new emails correctly without any formal training, just from the pattern in the examples. ICL gives LLMs this same ability: demonstrate a pattern in the prompt, and the model follows it for new inputs.
Common Usage Examples
- Few-shot classification: "Translate to French: Hello → Bonjour, Thank you → Merci, Goodbye → ?" — model answers "Au revoir"
- JSON formatting: show 3 examples of input → structured JSON output; model follows for new inputs
- Many-shot: Gemini 1.5 with 1M context window performs few-shot on 1000s of examples — near fine-tuning quality
- Prompt sensitivity:
["positive", "negative"]labels vs["yes", "no"]— different formatting affects ICL accuracy - LangChain
FewShotPromptTemplate: automatically format examples into consistent prompt structure
Summary
In short: In-context learning lets LLMs adapt to new tasks from examples in the prompt without any weight updates — the capability that makes large models general-purpose tools redirectable in seconds.