← AI Terminology
TruthfulQA
TruthfulQA measures whether models avoid common human falsehoods and misconceptions when answering questions — truthfulness under adversarial phrasing.
It targets honesty failures distinct from raw knowledge tests.
It targets honesty failures distinct from raw knowledge tests.
Why It Matters in AI
Fluent models can confidently repeat myths. TruthfulQA stresses that failure mode and motivates alignment and factuality work. Useful complement to MMLU-style knowledge exams.
Key Points
| Aspect | Description |
|---|---|
| Use | Safety/quality model cards |
| Focus | Imitative falsehoods; truthful answers |
| Origin | Lin et al., 2022 |
| Failure | Sycophancy and popular misconceptions |
| Metrics | Truthfulness, informativeness (human/auto) |
| Related | Hallucination, calibration, RLHF |
Simple Analogy
Questions designed like tempting trivia myths — can you resist the popular wrong answer everyone repeats?
Common Usage Examples
- Evaluate base vs aligned models on TruthfulQA
- Pair with retrieval for factuality
- Watch regressions after preference tuning
- Human eval protocols from the paper
Summary
In short: TruthfulQA tests whether models resist common falsehoods — measuring truthfulness beyond mere fluency or exam knowledge.