Speech recognition benchmarks have a blind spot, and it’s not technical. The Open ASR Leaderboard’s error-rate metrics have gotten harder to game — private splits, benchmark-fitting analysis, better normalizers. But the leaderboard still reports one number: word error rate. That number hides something researchers have known for years. Commercial ASR systems are roughly twice as bad for Black speakers as white ones, and accuracy shifts further by gender, age, and accent. The test sets record what was said, not who said it.
Monsoon changes that. The new evaluation suite adds two languages to the leaderboard: Indian English and Hindi, the first Indic language on a multilingual tab that until now covered only European languages. Hindi alone has over half a billion speakers. The datasets are built for variance, not volume. Four splits span 4,888 speakers, each tagged with 12 demographic attributes including geography, age, gender, occupation, education, income band, and handset brand. The ten largest contributors account for at most 6.8% of total duration. More than half of speakers appear exactly once.
That design is deliberate. Most benchmarks are assembled from whatever audio is convenient — long recordings of a few talkers in quiet rooms. Monsoon inverts that. Contributors recorded on their own phones, indoors and outdoors, across hundreds of districts. The Indian English public split draws on 428 native districts across 30 states and union territories. Prompts push speakers toward opinion, disagreement, and narration, which surfaces exactly the named entities and unrehearsed phrasing where ASR systems stumble.
Hindi created a unique problem: spelling variation. English normalizers can collapse most variants, but Hindi’s variants aren’t a fixed mapping between two conventions. So the Hindi sets ship with a lattice — for each span of transcript, a list of accepted spellings. That’s a meaningful shift in how ASR evaluation treats non-Latin-script languages, and it’s the kind of infrastructure decision that other Indic languages will likely borrow from. The public splits are available for self-scoring, with private splits withheld to limit benchmark-specific optimization.
