GPT-2 Can Supervise GPT-4? OpenAI's Surprising Alignment Gambit

OpenAI’s Superalignment team just dropped a paper that flips the script on a core AI safety problem: how do you control a system smarter than your entire team? Their answer, for now, is to use a dumb model to boss around a genius one. They set up an experiment where a GPT-2-level model acted as the ‘weak supervisor’ for GPT-4, trying to elicit its latent abilities without just teaching it to be as mediocre as its teacher.

The naive approach is a disaster. If you just fine-tune a strong model on a weak model’s crappy labels, you get a model that perfectly imitates all those errors—a powerful student that learned to fail exactly like its less capable instructor. But the researchers found a way around this by simply encouraging the strong model to be more confident, even to the point of confidently disagreeing with the supervisor when its vast internal knowledge says otherwise. The results were genuinely surprising: the GPT-2-supervised GPT-4 performed somewhere between a raw GPT-3 and GPT-3.5 on NLP tasks. That’s a massive recovery of capability from a supervisor that barely knows what it’s doing.

This isn’t a polished solution. The method completely face-planted on ChatGPT preference data, and the team is candid about the setup being a rough analogy for actual superhuman alignment. Future superhuman AIs might be far better at mimicking human flaws than GPT-4 is at mimicking GPT-2’s mistakes, which could make generalization much harder down the line. Still, the paper suggests that naive human feedback—like RLHF—could scale terribly to superhuman systems, and that better weak-to-strong generalization is at least a solvable research problem.

OpenAI is putting real money behind this hunch. They’re open-sourcing the code and launching a $10 million grants program aimed at grad students and academics to jump into the fray. The search is on for methods that scale, and for a deeper scientific understanding of when a strong model decides to listen to a weak signal versus when it correctly decides to go its own way.