Here’s something you don’t see every day: a three-line code change completely reshuffled the top 20 rankings on the most widely used AI leaderboard. Hugging Face just announced they’ve replaced the math evaluation parser for the Open LLM Leaderboard with a new tool called Math-Verify, then re-ran all 3,751 models ever submitted. The results are a stark reminder that how you measure performance can matter as much as the models themselves.
The old parser had a nasty habit of marking correct answers as wrong. Some models couldn’t follow the rigid “Final answer is [ANSWER]. I hope it is correct” formatting template, so their mathematically sound solutions were discarded. The SymPy-based extraction also choked on matrices, sets, and boxed notations — precisely the formats favored by several major model families. On average, models gained 4.66 points across the board after the fix, solving 61 more problems each. Algebra subsets saw the biggest jumps: 8.27 points for Algebra and 6.93 for Prealgebra, driven by better handling of multi-solution sets.
But the averages hide the real story. DeepSeek models nearly tripled their scores because the old evaluator couldn’t extract their \boxed{} answer format. Qwen models more than doubled theirs, finally matching the performance the team had self-reported. These weren’t incremental improvements — they were corrections of systematic underestimation that had persisted since June. Nvidia’s AceMath models now dominate the MATH-Hard leaderboard, with Qwen derivatives filling the spots right behind them.
This is a quiet but consequential moment for anyone using these scores to make decisions. The top four overall leaderboard positions didn’t move, but below them, some models jumped 200 or more places. It underscores a point I’ve been hammering for years: benchmark rankings are a product of the evaluation pipeline, not a pure measure of capability. When a 3-line parser swap can transform a model from also-ran to contender, the lesson isn’t that the models got smarter — it’s that we were measuring them wrong the whole time.
