Hugging Face kills black-box leaderboards with decentralized evals

Hugging Face just took a swing at one of AI’s dirtiest open secrets: benchmark scores you can’t trust. The platform announced that dataset repos can now host leaderboards, model repos store their own eval scores, and — here’s the real shift — anyone in the community can submit results via pull request. No more waiting for model authors to bless the numbers.

The initial rollout covers four benchmarks. MMLU-Pro, GPQA, and HLE are already live, with more promised as the feature matures. Each benchmark defines its evaluation spec through an eval.yaml file based on the Inspect AI format, which means the task definition is public and reproducible — at least in theory. Verified badges will flag results that can be independently reproduced, a small but meaningful hedge against cherry-picked numbers.

Let’s be clear about what this doesn’t fix. Benchmark saturation isn’t going anywhere. MMLU is above 91%. GSM8K is over 94%. Models that ace these tests still hallucinate on multi-step tasks and can’t write production code. Training on test sets will continue. What changes is visibility: you’ll be able to see who submitted a score, when, and what source it links to — a paper, a model card, an eval log. The entire history sits in Git, so score changes leave a trail.

This feels less like a technical breakthrough and more like an overdue governance move. By making evaluation a public, messy, version-controlled process, Hugging Face is betting that transparency will do what centralized leaderboards couldn’t — surface which numbers actually hold up. The real test is whether the community bothers to submit honest results, or if we just get a new flavor of benchmark gaming with a paper trail attached.