← AI Terminology
Data Poisoning
Data poisoning is an adversarial attack where an attacker manipulates the training data of a machine learning model — injecting malicious examples that cause the trained model to behave incorrectly in targeted or untargeted ways.
It exploits the fact that ML models learn entirely from their training data.
It exploits the fact that ML models learn entirely from their training data.
Why It Matters in AI
As AI is deployed in safety-critical applications and trained on web-scale data scraped from sources attackers can influence, data poisoning becomes a serious real-world threat. An attacker who can inject even a small fraction of training examples can insert backdoors, cause targeted misclassification, or degrade overall performance. Models trained on internet data (including LLMs) are particularly exposed because training data curation is imperfect at scale.
Key Points
| Aspect | Description |
|---|---|
| Defence | Data sanitisation, certified defences, anomaly detection on training loss, data provenance |
| Integrity | Targeted: cause specific misclassifications (e.g. spam flagged as legitimate) |
| Availability | Untargeted: degrade overall model performance — reduce accuracy across the board |
| LLM exposure | Web-scraped training data — attackers can publish poisoned content that gets scraped |
| Supply chain | Poisoning pre-trained models on HuggingFace Hub — downstream users inherit the backdoor |
| Backdoor attack | Clean-label: insert a trigger pattern; model behaves normally except when trigger is present |
Simple Analogy
Imagine a student who knows the upcoming exam will draw from a public question bank — and secretly plants subtly wrong answers in some questions. Students who study from that bank learn the wrong information. Data poisoning does the same to ML models: corrupt a fraction of the training data and the model learns what the attacker wants.
Common Usage Examples
- Badnets backdoor: add a small trigger pattern (e.g. a white square) to 1% of training images labelled as target class
- LLM poisoning: publish web content containing rare trigger phrases with biased outputs
CleanLablibrary: detect label errors and potentially poisoned examples via confident learning- Supply chain attack: modified model weights uploaded to HuggingFace with hidden backdoor behavior
- Defence: influence function analysis to identify which training examples most affect a prediction
Summary
In short: Data poisoning corrupts the training data to corrupt the model — a stealthy attack that is hard to detect and especially dangerous for models trained on unverified web-scale data.