← AI Terminology
Interpretability
Interpretability is the degree to which a human can understand why an AI model produced a specific output — examining internal mechanisms (weights, activations, attention patterns) rather than just input-output behaviour.
It is distinct from explainability, which provides post-hoc approximations; interpretability aims for mechanistic understanding.
It is distinct from explainability, which provides post-hoc approximations; interpretability aims for mechanistic understanding.
Why It Matters in AI
Black-box models can be accurate but untrustworthy: they may exploit spurious correlations, encode bias, or fail unpredictably on edge cases. Interpretability research (mechanistic interpretability, circuit analysis, probing) lets engineers verify what a model has actually learned. For high-stakes domains — medical diagnosis, credit decisions, autonomous systems — regulators increasingly require interpretable or explainable models.
Key Points
| Aspect | Description |
|---|---|
| Tools | TransformerLens (Anthropic), BertViz, Captum (PyTorch), Inseq |
| Probing | Train a lightweight classifier on internal activations to test what information is encoded |
| Regulation | EU AI Act and GDPR "right to explanation" drive demand for interpretable models in EU |
| SHAP / LIME | Post-hoc approximation methods — technically explainability, not mechanistic interpretability |
| Attention maps | Visualise which tokens the model attends to — informative but not reliably causal |
| Mechanistic interp | Reverse-engineering specific model circuits (e.g. how GPT-2 detects indirect objects) |
Simple Analogy
A judge who can explain every step of their reasoning in plain language (interpretable) vs. a judge who gives a verdict but can only offer a vague summary of why (explainable). Interpretability demands the full internal chain of logic — not a post-hoc rationalisation.
Common Usage Examples
transformer_lens.HookedTransformer— Anthropic's library for mechanistic interpretability of transformer models- Activation patching: replace activations at one layer with those from a different input to trace causal paths
captum.attr.IntegratedGradients(model).attribute(input)— gradient-based attribution in PyTorch- Probing:
LinearProbe(train_data=activations_layer_12, labels=pos_tags)— tests if syntax is encoded - Anthropic's "Towards Monosemanticity" paper: sparse autoencoders to find interpretable features in MLP layers
Summary
In short: Interpretability is the science of understanding what AI models actually compute internally — essential for trust, safety, and debugging in high-stakes applications.