OpenAI trained GPT-4 to catch GPT-4's own mistakes

OpenAI researchers have essentially taught GPT-4 to police itself. They’ve developed a new model, CriticGPT, specifically designed to write critiques that help human trainers spot errors in ChatGPT’s code output. The immediate goal isn’t to replace human reviewers but to give them a sharp-eyed assistant, a move that feels less like a sci-fi fantasy and more like a necessary repair job on the standard AI training process.

That process, known as Reinforcement Learning from Human Feedback (RLHF), has a growing flaw: as models like GPT-4 get more sophisticated, their mistakes get subtler. When a system is smarter than the person evaluating it, the feedback loop breaks down. CriticGPT is designed to bridge that gap. In practical terms, the results are tangible. When human trainers got help from CriticGPT, they outperformed their unassisted colleagues 60% of the time. The critiques produced by the human-AI team were preferred over solo human work more than 60% of the time by a second reviewer.

Training the critic model involved a clever bit of role-playing. Trainers intentionally inserted bugs into code written by ChatGPT, then wrote sample feedback as if they had discovered the error themselves. This data taught CriticGPT to generate more focused critiques. The model proved particularly good at cutting down on ‘nitpicks’—those unhelpful minor complaints—and hallucinating fewer problems than a standard ChatGPT instance. On naturally occurring bugs, trainers preferred CriticGPT’s analysis in 63% of cases.

But this isn’t a solved problem. The approach currently works best on short code snippets where errors are localized in a single spot, not scattered across a complex response. OpenAI is candid that for extremely convoluted tasks, even an expert armed with a tool like CriticGPT will struggle. The research sketches out the path to aligning future AI agents that might be too competent for humans to directly supervise. For now, it’s a smart patch on a fundamental limitation, and the plan is to fold these critic models directly into OpenAI’s RLHF pipeline to produce better training data.