NVIDIA's Conformer-LLM hybrid tops ASR accuracy, but open-source still trails in long-form audio

The Open ASR Leaderboard just got a major expansion, adding multilingual and long-form transcription tracks that reveal a fractured landscape. After benchmarking over 60 models from 18 organizations, a clear pattern has emerged: the best accuracy comes from pairing a Conformer encoder with a large language model decoder. NVIDIA’s Canary-Qwen-2.5B, IBM’s Granite-Speech-3.3-8B, and Microsoft’s Phi-4-Multimodal-Instruct now sit at the top of the English charts with the lowest word error rates. That LLM reasoning boost is real, but it’s not free.

Speed is where things get interesting. Those top-performing LLM-based decoders are sluggish compared to CTC and TDT alternatives, which deliver 10 to 100 times the throughput. This is the classic engineering tradeoff, and the numbers are stark. For long-form jobs like meeting transcription, NVIDIA’s Parakeet CTC 1.1B hits an inverse real-time factor of 2793.75—that’s wildly fast—while OpenAI’s Whisper Large v3 limps along at 68.56. The WER difference is negligible (6.68 vs. 6.43), but Parakeet’s speed comes with a catch: it’s English-only.

On the multilingual front, Whisper Large v3 remains the safe bet for broad coverage across 99 languages. But specialization is the real story. Fine-tuned variants like Distil-Whisper and CrisperWhisper stomp the original on English benchmarks, sacrificing linguistic range for precision. Self-supervised systems like Meta’s MMS support over 1,000 languages but can’t match dedicated models on accuracy. As one of the preprint authors notes, focusing on English tends to reduce multilingual coverage, a classic generalization-specialization tradeoff.

Despite all this progress, closed-source systems still hold the lead in long-form transcription. Podcasts, lectures, and meetings remain stubbornly difficult, likely due to proprietary chunking strategies and domain-specific tuning. The preprint, available on Hugging Face, frames this gap as one of the most exciting frontiers for open-source innovation. The leaderboard is also expanding its language coverage beyond its current five and accepting community contributions via GitHub, mirroring the approach of specialized benchmarks like the Open Universal Arabic ASR Leaderboard.