GPT-3 Learns to Say 'I Don't Know' Without Peeking at Its Own Code

Here’s something genuinely new: researchers at OpenAI have taught GPT-3 to tell you how confident it is in its own answers using plain English, without ever letting the model check its internal probability scores. Stephanie Lin, Jacob Hilton, and Owain Evans showed that the model can generate answers paired with natural language confidence levels—think “90% confidence” or “high confidence”—and those verbalized probabilities actually match reality. It’s not just mimicking human uncertainty either. The model seems sensitive to what it genuinely doesn’t know.

To test this, the team built a new benchmark called CalibratedMath, a suite of math problems designed to probe how well a model’s stated confidence lines up with its accuracy. The results were surprisingly solid. The model stayed moderately calibrated even when the questions shifted away from what it was trained on. That’s a big deal because most AI systems fall apart when you yank them out of their comfort zone.

What’s really clever is how they compared this verbalized uncertainty to the old-school method of extracting confidence from model logits. Both approaches generalized under distribution shift, but the natural language route opens up a much more intuitive way for users to interact with AI. Instead of staring at a raw probability number, you get a conversational signal.

The paper argues this ability hinges on latent representations GPT-3 picked up during pre-training—representations that correlate with epistemic uncertainty, the kind of doubt that comes from not having enough knowledge rather than just noisy data. That suggests the model built some internal sense of what it doesn’t know long before anyone asked it to confess. Whether this holds up in messier, real-world domains beyond clean math problems remains an open question, but for now, it’s a rare example of a language model getting calibration right without cheating.