Hugging Face locks down new speech datasets to stop AI benchmark gaming

The Open ASR Leaderboard just got its first taste of private test data — and it’s a direct shot at the subtle art of ‘benchmaxxing.’ Hugging Face has partnered with Appen Inc. and DataoceanAI to curate high-quality English speech recognition datasets covering scripted and conversational audio across multiple accents. The catch? Model developers can’t see or download them. This isn’t secrecy for its own sake; it’s a calculated move to prevent the kind of benchmark-specific optimization where a model’s score looks brilliant on a public leaderboard but crumbles when faced with real-world audio. Since launching in September 2023, the leaderboard has pulled in over 710,000 visits, making it a central scoreboard for the ASR community — and a tempting target for gaming.

The new private data introduces a more nuanced scorecard without exposing the test questions. A new ‘Private data’ tab on the leaderboard will show aggregate metrics — like average word error rate for conversational speech versus scripted, or US accents versus non-US — but pointedly refuses to break out scores by individual dataset or data provider. That granularity, the team argues, would just paint a new bullseye for hyper-optimizers. ‘We intentionally do not provide a score on each split, to avoid model developers from boosting their score with a specific data provider or accent,’ the announcement states.

The move balances two forces that have always been in tension: standardization and openness. Both make benchmarking useful, but both also make it exploitable. By keeping these particular datasets private, Hugging Face hopes to offer a trustworthiness signal that public benchmarks alone can’t provide. The default average WER on the leaderboard will still be computed solely on the existing public datasets, with the private metrics available as an optional toggle for those who want to peek behind the curtain.

This update also reinforces a core finding from the team’s earlier report: there is no single ‘catch-all’ ASR model. Some excel on American English but stumble on diverse accents; others prioritize speed over accuracy on conversational audio. The goal isn’t to crown one king, but to show where each model actually earns its keep — and where it falls apart. As evaluation becomes a higher-stakes game, keeping some cards close to the chest might be the only way to ensure the scores on the board mean something when the microphone is live.