OpenAI Audit Reveals 30% of Coding Benchmarks Are Broken

OpenAI conducted a deep audit of SWE-Bench Pro, a popular coding benchmark used to evaluate AI models’ software engineering skills, and discovered that roughly 30% of its tasks are flawed. The problems fall into four categories: overly strict tests that reject valid solutions, vague prompts that don’t give models enough context, tests with low coverage that let incomplete fixes slip through, and misleading prompts that point models in the wrong direction. This matters because as AI models rapidly improve—frontier models jumped from a 23.3% to 80.3% pass rate in just eight months—relying on broken benchmarks can create a false sense of capability, skewing safety assessments and research priorities. OpenAI now advises developers to carefully examine their results and has built a quality assurance pipeline to catch these issues. Why it matters: This audit underscores a growing challenge in AI evaluation—benchmarks are only as good as their data, and as models get smarter, the benchmarks themselves need to be held to a higher standard.