NVIDIA's 11B voice model kills the ASR-LLM-TTS chain — and brings tool calling

NVIDIA dropped an 11-billion-parameter model that finally collapses the clunky ASR-to-LLM-to-TTS pipeline into a single network. NemotronLabs VoiceChat 11B does streaming speech understanding and generation in one go, and the numbers are genuinely attention-grabbing: 448 milliseconds of smooth turn-taking latency on the Full-Duplex-Bench 1.0. For anyone who’s built voice agents, you know that sub-half-second responsiveness is the line between “this feels like magic” and “why am I waiting.”

But the headline feature here isn’t just speed. It’s tool calling. This is the first open full-duplex model that can fire off an API call on a separate output channel while the conversation keeps moving. When the model decides it needs external data, it emits a block and immediately speaks an operator-defined “on-hold” line so the user isn’t met with dead air. Think of it as the voice equivalent of a web app’s loading spinner, except it’s a casual “let me check that for you” delivered in real-time while the API runs. The model listens even while it’s speaking, so barge-in works — it yielded to user interruption with a take-over rate of 1.00 at 480 milliseconds.

Architecturally, it’s a hybrid Mamba/Transformer built from three existing NVIDIA components: a Fast Conformer encoder from Nemotron-Speech-Streaming, the Nemotron Nano v2 LLM backbone, and an NVIDIA TTS decoder. Training soaked up roughly 550,000 hours of audio. The outputs are tri-modal — agent audio, agent text, and a running transcription of the user — which makes debugging and logging far less painful than with black-box cascaded systems.

Now the reality check. NVIDIA is explicit that this checkpoint is “ready for research purposes only,” and the documented failure modes are exactly what you’d expect from a bleeding-edge speech model. There’s a two-minute audio context ceiling, and after several turns the model can degrade into non-recoverable gibberish or start talking to itself after a turn ends. Dropped words in user transcription are a known issue. You also need one GPU with at least 80 GB of VRAM — A100, H100, or equivalent — and there’s no hosted API or inference provider serving the model. That keeps it firmly in the hands of teams with hardware budgets: AI-native startups, enterprise R&D labs, GPU cloud providers, and university speech groups. The tool-calling benchmarks tell a mixed story too. On the BFCL-v3 spoken tool-calling benchmark it hit 56.1% average, with parallel-multiple tool calls scraping just 27.5%. NVIDIA recommends a five-tool ceiling and warns that simultaneous tool calls aren’t reliable.

Still, ranking #2 among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0 puts this in a category with very few occupants. For contact center platforms, automotive assistants, drive-thru ordering systems, or any team building voice front-ends over internal APIs, this is the kind of model you prototype with today and watch closely for the production-ready checkpoint. The permissive OpenMDW-1.1 license means the weights are yours to tinker with. Just budget for that GPU.