← AI Terminology
Red Teaming
Red teaming in AI is the systematic adversarial testing of AI models by a dedicated team (or automated system) that attempts to elicit harmful, unsafe, or unintended outputs — identifying vulnerabilities in safety training before deployment.
The term comes from military wargaming where a "red team" simulates adversarial attacks.
The term comes from military wargaming where a "red team" simulates adversarial attacks.
Why It Matters in AI
Safety training (RLHF, Constitutional AI) cannot anticipate every attack vector — red teams find the gaps. Frontier AI labs (Anthropic, OpenAI, Google DeepMind, Meta) have dedicated red teams that spend months before each model release probing for jailbreaks, harmful content generation, bias amplification, CSAM generation, bioweapons uplift, and other critical failure modes. External red teaming (bug bounties, third-party audits) is increasingly required by regulation (EU AI Act) for high-risk AI systems.
Key Points
| Aspect | Description |
|---|---|
| Output | Red team reports → safety training improvements → re-evaluation — iterative cycle |
| Domains | Harmful content, CBRN (chemical/bio/radiation/nuclear) uplift, CSAM, cyberattacks, manipulation |
| Regulation | EU AI Act requires red teaming for GPAI models above 10^25 FLOPs and all GPAI with systemic risk |
| External red team | Independent security researchers, bounty programs — diverse attack perspectives |
| Internal red team | Company's own adversarial safety team — broad, systematic, before release |
| Automated red team | LLM-generated adversarial prompts at scale — RL-based attacker trained to find jailbreaks |
Simple Analogy
A bank hiring professional thieves to test its vault before opening: the thieves (red team) probe every lock, alarm, and procedure to find weaknesses. If they succeed, the bank fixes the vulnerabilities before real criminals try. AI red teaming is the same — hired adversaries find what real bad actors would find first.
Common Usage Examples
- Anthropic: red team findings documented in model cards and safety reports for Claude models
- OpenAI: GPT-4 System Card includes red team findings across 50+ risk categories
aart(Google): Automated Adversarial Red Teaming — LLM-based attack generation at scalegaraklibrary: open-source LLM vulnerability scanner — tests jailbreaks, hallucinations, toxicity- Bug bounty: OpenAI/Anthropic programs pay external researchers for novel safety vulnerabilities
Summary
In short: Red teaming systematically attacks AI models before deployment to find safety vulnerabilities — the adversarial testing practice that identifies jailbreaks and harmful outputs so they can be patched before reaching real users.