OpenAI's Codex solves 70% of coding puzzles by guessing 100 times

OpenAI has formally introduced Codex, the code-generating descendant of GPT-3 that’s been powering GitHub Copilot behind the scenes, and the performance gap is staggering. In a newly released benchmark called HumanEval — a set of 164 hand-written programming problems designed to test functional correctness from docstrings — vanilla GPT-3 solved exactly zero. Not a single one. GPT-J managed 11.4%. Codex? It hit 28.8% on the first try, a number that climbs dramatically when you let the model take multiple swings at the same problem.

That’s where things get genuinely interesting. The researchers discovered that repeated sampling is a surprisingly effective strategy. Generate 100 candidate solutions per problem and pick the one that passes the unit tests, and Codex suddenly clears 70.2% of the benchmark. It’s a brute-force approach that works because the model produces enough variety in its outputs — most of them wrong, but different flavors of wrong — that one often stumbles into correctness. This isn’t elegant reasoning. It’s statistical persistence. But it works, and it raises uncomfortable questions about how we evaluate AI systems that look smarter in aggregate than any single output would suggest.

The paper doesn’t shy away from limitations. Codex struggles when docstrings describe long chains of operations — the kind of multi-step reasoning a junior developer might sketch on a whiteboard. It also fumbles at binding operations to variables, a fundamental skill that human programmers internalize in their first weeks of learning. These aren’t exotic edge cases. They’re bread-and-butter programming tasks, and the model’s brittleness here suggests we’re still far from anything resembling robust code understanding.

Perhaps most notable is what the paper discusses beyond the benchmark numbers: safety, security, and economics. Code generation at this scale creates genuinely novel risks, from accidentally suggesting insecure code to disrupting labor markets that weren’t expecting automation to arrive quite this fast. OpenAI released the paper alongside a production version of Codex that’s distinct from the research model, and the May 2025 update notes the launch of a cloud-based agent that can juggle many tasks in parallel. The era of AI pair programming didn’t start with this paper — Copilot’s been in developers’ editors for a while — but the data is now public, and the trajectory is clear. The question isn’t whether these models can write code. It’s what happens when they start writing it faster than we can review it.