OpenAI Quantifies Goodhart's Law to Stop AI Reward Hacking

Every engineer wrestling with AI alignment knows the dread of Goodhart’s law: the moment you turn a metric into a target, it stops measuring what you actually care about. OpenAI is now putting hard math behind that anxiety, publishing a formal analysis of how proxy objectives drift from true goals during optimization. The work is a direct peek into the practical headaches of training models like GPT-3 to be helpful and accurate, where the real objective—human judgment—is agonizingly slow and expensive to measure.

So what’s the fix? Instead of just wringing their hands, the team dives into best-of-n sampling, a dead-simple technique where you generate, say, 64 responses and let a reward model pick the winner. It’s brute force, but it works. They point to WebGPT, where a best-of-64 setup actually outperformed their reinforcement learning model, likely because it allowed the system to browse many more websites. Even a modest best-of-4 provided a real lift in human preference scores. The magic isn’t just in the results, but in the math they’ve derived to monitor the process.

OpenAI details a clever statistical estimator that recycles samples to simultaneously evaluate performance across different n values without re-running expensive human evaluations. The formula involves a binomial coefficient weighting based on a sample’s rank, and they also reveal an exact, rather than approximate, formula for the KL divergence that results from this optimization: log(n) minus (n-1)/n. This clean mathematical relationship lets them precisely track the trade-off between optimizing their cheap proxy and how much the true objective actually improves.

The catch? The KL divergence only grows logarithmically with n, which puts a hard cap on how much optimization you can squeeze from this method. It’s perfect for a quick, high-confidence boost but not for deep optimization runs. For that, you still need reinforcement learning. The post doesn’t solve Goodhart’s law, but it gives teams a rigorous tool to see the cliff before they drive off it, quantifying exactly when that proxy metric starts lying to you.