AI Critics Help Humans Spot 50% More Flaws in Summaries

Here’s a finding that should matter to anyone worried about AI systems learning to deceive us: when humans are shown AI-generated critiques, they get significantly better at spotting errors they’d otherwise miss. OpenAI’s latest research demonstrates that critique-writing models can boost human flaw detection rates by 50% on model-written summaries, and for deliberately misleading text, the improvement is even starker — humans caught the intended flaw 45% of the time with assistance versus just 27% without it.

The setup was straightforward. Researchers trained language models using supervised learning to write critiques of summaries from short stories and Wikipedia articles. Then they showed labelers eight model-written critiques per summary, with a control group getting no help at all. The summaries came from three sources: AI-written, human-written, and human-written summaries specifically designed to contain subtle, tricky flaws. The assisted group consistently outperformed their unassisted peers.

Scale matters here in a specific way. Larger models proved better at self-critiquing than smaller ones, and they could actually use those critiques to improve their own outputs. Smaller models couldn’t pull off that feedback loop. But there’s an uncomfortable wrinkle: models are better at knowing something is wrong than they are at explaining what’s wrong. That gap between discrimination and articulation didn’t shrink as models got bigger. The researchers flagged this as a priority for alignment work — if a model can detect a flaw but can’t communicate it, we’ve still got a supervision problem.

Let’s not overstate things. Summarization isn’t exactly a Herculean task — humans can evaluate one in about 10 minutes. The real test will be whether this approach holds up on genuinely difficult problems, like auditing a massive codebase or fact-checking a book-length argument. But the direction is promising. The long game, as OpenAI frames it, is building assistants that handle all the cognitive heavy lifting of evaluation so humans can focus on stating what they actually want. Whether we get there without the articulation gap biting us is an open question worth tracking.