OpenAI Finds SWE-bench Is Broken—So They're Fixing It

SWE-bench has been the gold standard for proving your AI can code like a real engineer. Top models barely scraped 20% on the full set. But OpenAI’s Preparedness team, which needs reliable metrics to gauge when models get dangerously autonomous, started poking at the benchmark’s internals and didn’t like what they found.

Working with the original authors, they’re releasing SWE-bench Verified, a human-validated subset designed to fix what they see as systematic underestimation of AI coding skills. The core problems are exactly the kind of stuff that makes engineers roll their eyes at academic benchmarks. Unit tests are often so specific they reject functionally correct solutions—like demanding a specific deprecation warning the original GitHub issue never mentioned. Issue descriptions can be maddeningly vague, leaving models to guess at what “resolve the issue” actually means. And sometimes the development environments themselves are flaky, causing tests to fail regardless of the fix.

OpenAI published a concrete example from scikit-learn that illustrates the absurdity. An issue reports that a function’s copy parameter is ignored. A reasonable fix might just make the parameter work. But the SWE-bench test demands the solution raise a DeprecationWarning—a requirement nowhere in the problem statement. Even if an agent correctly deduced the need for a warning, the test checks for specific message text it could never know. These aren’t edge cases. They’re systematic failures that silently penalize capable models.

This matters beyond just fairer leaderboard scores. OpenAI’s Preparedness Framework ties autonomous coding ability directly to risk levels for model deployment. If the benchmark is noisy, their safety assessments are too. SWE-bench Verified isn’t a new benchmark—it’s a cleanup operation on one of the most consequential evaluations in the field. Whether other labs adopt it or stick with the original will be a quiet test of how seriously the industry takes measurement integrity.