← AI Terminology
Pruning
Pruning is a model compression technique that removes parameters (weights, neurons, attention heads, or entire layers) from a trained neural network that contribute least to the output — reducing model size and inference cost while preserving as much accuracy as possible.
It is one of the three main compression methods alongside quantisation and knowledge distillation.
It is one of the three main compression methods alongside quantisation and knowledge distillation.
Why It Matters in AI
Large neural networks are massively over-parameterised: the lottery ticket hypothesis showed that sparse subnetworks embedded in large models can match full-network accuracy when trained appropriately. Pruning removes the redundant parameters, reducing model size (fewer bytes to store and transfer), inference FLOPs (fewer operations per pass), and sometimes latency — enabling deployment on edge devices. Structured pruning (removing whole channels or heads) directly reduces hardware operations; unstructured pruning requires sparse hardware to realise speedups.
Key Points
| Aspect | Description |
|---|---|
| Structured | Remove whole neurons, attention heads, or layers — directly reduces FLOPs on any hardware |
| Unstructured | Remove individual weights (set to zero) — high sparsity possible; requires sparse compute support |
| After pruning | Fine-tuning the pruned model restores most accuracy — prune → fine-tune cycle |
| Lottery ticket | Frankle & Carlin (2019): sparse "winning ticket" subnetworks exist from random initialisation |
| Gradual pruning | Prune incrementally during training (or fine-tuning) with sparse masks — better accuracy |
| Magnitude pruning | Remove weights with smallest absolute value — simple heuristic; surprisingly effective |
Simple Analogy
Trimming a hedge: cut the branches (weights) that contribute least to the shape without destroying the overall form. An aggressive trim (high sparsity) leaves a smaller, efficient hedge; a conservative trim preserves more but reduces less. After trimming, the plant fills in (fine-tuning recovers accuracy).
Common Usage Examples
torch.nn.utils.prune.l1_unstructured(layer, name='weight', amount=0.3)— prune 30% of weights by L1 normtorch.nn.utils.prune.global_unstructured(parameters, pruning_method=L1Unstructured, amount=0.5)- Structured: prune attention heads with low importance scores —
SparseGPT(2023) prunes 50% of Llama weights SparseMLlibrary: iterative magnitude pruning with fine-tuning for production model compression- Sparsity masks:
model.weight.data[mask == 0] = 0— zero out pruned weights for inference
Summary
In short: Pruning removes the least-important parameters from a trained network — reducing model size and inference cost while preserving accuracy, with structured pruning delivering direct hardware speedups on any device.