🧠 Why do language models hallucinate? OpenAI researchers finally have an answer.
For years, AI hallucinations were treated as a mysterious flaw. Now, a new OpenAI paper points to the root cause: training objectives reward guessing instead of acknowledging uncertainty.
📊 The problem in numbers:
- Models learn that trying to answer — even if wrong — produces better benchmark performance than saying “I don’t know.”
- A model with a 75% error rate can outperform another with just 1% abstention rate, simply due to lucky guesses.
🔍 The two main causes:
- Pre-training: predicting the next word doesn’t teach models to recognize their own limits.
- Post-training: benchmarks indirectly penalize abstaining, rewarding the number of correct answers over epistemic honesty.
🛠️ The proposed solution: Redesign popular benchmarks (GPQA, MMLU, SWE-bench) to explicitly penalize incorrect answers and reward “I don’t know” when appropriate.
💡 In a nutshell#
Imagine an exam where guessing is always worth it because wrong answers carry no penalty. What would you do? You’d guess everything. AI models learned exactly that: it’s more “profitable” to make up a plausible answer than to admit ignorance. The solution is to change the exam rules so that lying has a real cost.
⚠️ One open problem remains: out-of-distribution (OOD) situations, where the model faces cases it never encountered during training. That, for now, has no simple solution.
More information at the link 👇
