← AI Terminology
Explainable AI - XAI
Explainable AI (XAI) is the field of methods and practices that make AI model predictions understandable to humans — explaining why a model made a specific decision, which features drove it, and how confident it is.
It is increasingly required by regulation (EU AI Act, GDPR's right to explanation) and demanded by users in high-stakes decisions.
It is increasingly required by regulation (EU AI Act, GDPR's right to explanation) and demanded by users in high-stakes decisions.
Why It Matters in AI
A black-box model that says "loan denied" gives the applicant no recourse and the operator no way to audit for bias. XAI methods — SHAP, LIME, attention visualisation, saliency maps — open the black box enough to verify, debug, and trust AI decisions. Regulators are mandating explainability for high-risk AI systems; XAI tools have become a standard component of ML deployment pipelines.
Key Points
| Aspect | Description |
|---|---|
| LIME | Approximates a black-box model locally with an interpretable model — works for any model |
| SHAP | Assigns each feature a contribution value (Shapley value) — model-agnostic, theoretically grounded |
| Intrinsic | Inherently interpretable models: linear regression, decision trees, rule-based systems |
| Regulatory | GDPR Article 22: right to meaningful explanation for automated decisions |
| Saliency maps | Gradient-based: pixels/tokens most responsible for the prediction |
| Attention maps | Visualise where the model "looks" in an image or text — limited as an explanation (debated) |
Simple Analogy
A loan officer who can say "I denied this loan because debt-to-income ratio was 45% (over our 40% threshold) and employment history was only 6 months" is explainable. A neural network that says "denied" with no reason is not — XAI retrofits that explanation capability onto models that don't have it natively.
Common Usage Examples
shap.Explainer(model)(X_test)— SHAP values for any model;shap.waterfall_plot()for individual predictionslime.lime_tabular.LimeTabularExplainer— local surrogate model for a single prediction- Grad-CAM: saliency map showing which image regions activated the classification decision
- Integrated Gradients:
captum.attr.IntegratedGradients(model).attribute(input, target=class_idx) - SHAP summary plots:
shap.summary_plot(shap_values, X)— global feature importance across all predictions
Summary
In short: XAI makes AI decisions interpretable — essential for debugging bias, building user trust, satisfying regulators, and making high-stakes automated decisions auditable.