OpenAI Says the Way We Score AI Actually Rewards Lying

OpenAI has a new theory for why chatbots keep making stuff up, and it points the finger squarely at the people building them—specifically, at how they grade their own creations. In a new research paper, the company argues that language models hallucinate because standard training and evaluation procedures inadvertently reward guessing over admitting uncertainty. The core problem is a fixation on raw accuracy. When a model is scored solely on whether its final answer is correct, it learns that a wild guess with a 1-in-365 shot is statistically better than a responsible ‘I don’t know,’ which scores a guaranteed zero. The paper offers a bluntly concrete example: when asked for the PhD dissertation title of one of the paper’s own authors, a widely used chatbot confidently produced three different, entirely fabricated answers. The same thing happened with his birthday—three wrong dates, delivered without a flicker of doubt.

This isn’t just a theoretical gripe. The paper uses data from the GPT-5 system card to show that while an older o4-mini model might technically edge out a newer one on a simplistic accuracy benchmark like SimpleQA, its error rate is significantly higher. It’s a classic case of gaming the test. The model that strategically guesses looks better on the leaderboard, even though it’s hallucinating more often. OpenAI’s position, as stated in its Model Spec, is that a confident error is worse than an abstention. Yet the industry’s dominant scoreboards continue to force a binary choice between right and wrong, completely ignoring the crucial third category of responsibly admitting ignorance.

The fix OpenAI proposes is straightforward in theory but disruptive in practice: overhaul the scoring. They suggest penalizing confident errors more harshly than expressions of uncertainty and giving partial credit for appropriate abstentions. The idea mirrors the negative marking long used on some standardized tests to discourage blind guessing. What makes their argument pointed is the claim that it’s not enough to simply add a few new, uncertainty-aware tests on the side of existing evaluations. The main leaderboards that developers obsess over, the ones that drive headlines and funding, have to change. If the primary scoreboards keep rewarding lucky guesses with higher accuracy numbers, models will keep being optimized to guess.

The paper also touches on the stubborn nature of these factual errors, noting that large pretrained models rarely make other basic mistakes like spelling errors, making their propensity for fabricating highly specific, plausible-sounding falsehoods a distinct and deeply embedded flaw. The implication is clear: the hallucination problem is less a temporary bug and more a logical consequence of an industry that’s been grading a multiple-choice test with only one real rule—always fill in the bubble, even if you’re just picking C.