ICML 2026 mass audit: 23% of papers had claims falsified or contested

The ICML 2026 Open Reproductions challenge just did something unprecedented: it crowdsourced the verification of an entire machine learning conference. From July 15 to August 2, 2026, participants used coding agents like Claude Code, Codex, and Cursor to read papers, write reproduction code, and run experiments. The result? A staggering 2,962 cloud jobs launched, 2,153 papers examined, and a sobering verdict on the state of AI research.

Of the papers examined, 51% had at least one claim independently verified, with 266 fully reproduced and 3,978 individual claims confirmed. But the other number is the one that should keep researchers up at night: 23% of papers had at least one claim falsified or contested. That’s 496 papers. Forty-nine had every single claim falsified. Another 242 papers saw independent teams reach opposite conclusions on the same claims—a reminder that reproducibility is not binary but adversarial.

The challenge surfaced some spectacular failures hiding in plain sight. One ICML 2026 spotlight paper on learning-augmented paging had a reviewer who admitted, “My low confidence score is because I did not check all the proofs carefully.” When the community finally did check, they found the paper’s claimed robustness guarantee was wrong. A participant located the exact proof step that breaks, and the organizers’ own re-implementation confirmed the error at roughly nine sigma. Another paper’s theorem about attention and Frank-Wolfe collapsed after step 224, with three independent teams finding counterexamples at different horizons—explaining why everyone else’s finite-horizon checks missed it.

The backdrop here is scale. ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year. Reviewers are volunteers who can’t possibly verify every proof. But the same agents flooding conferences with papers can also audit them—checking a paper that used to cost a reviewer a weekend now takes an afternoon, in parallel, thousands of times over. The question is whether the community will institutionalize this kind of adversarial verification, or whether it remains a one-off hackathon spectacle.