NVIDIA's Nemotron 3.5 Lightning cuts agent costs by running on a single GPU

The dirty secret of AI agents is that most of the work isn’t reasoning—it’s plumbing. Tool calls, checking results, and handing off tasks between sub-agents eat up the vast majority of tokens, and routing every one of those mundane steps to a massive frontier model is like taking a Ferrari to check your mailbox. NVIDIA’s response is a two-part release that directly targets this structural inefficiency, and it’s more interesting than the usual spec-sheet race.

The headliner is Nemotron 3.5 Lightning. This is a 30-billion-parameter model built on a hybrid Mamba-2 and attention architecture, but its key trick is being a mixture-of-experts design where only 3 billion parameters are active at any time. The result is a model explicitly built for high-volume execution tasks, with a context window stretching to 1 million tokens. NVIDIA reports up to a 4x increase in output speed compared to models of a similar size, and on a benchmark of 10,000 PinchBench tasks, it completed the gauntlet 30% faster than Qwen3.6 35B while matching its accuracy. It does this through a combination of speculative decoding—using dedicated draft models called DSpark and DFlash to predict multiple tokens at once—and native NVFP4 quantization.

What makes this genuinely newsworthy for builders isn’t just the performance, but the deployment profile. This model runs on a single GPU, specifically called out as deployable on a single DGX Spark or H100. That collapses the barrier between a solo developer prototyping on a workstation and an enterprise running on-premises servers. Companies like CrowdStrike, Harvey, and CodeRabbit are already customizing it for their specific domains, but the open-source nature under the OpenMDW-1.1 license, complete with training data and recipes, means the long tail of startups and tinkerers can get to work without a cloud bill that requires venture funding.

The second artifact, NeMo Switchyard, is the routing intelligence that makes this ecosystem practical. It’s an open-source library that decides which model should handle each step of an agent’s workflow. One of its routers, tested by LangChain across 145 multi-turn tasks, routed between Lightning and Claude Opus 4.8 and slashed costs by 74% while sending only 7% of calls to the more expensive frontier model, accepting a roughly 6-point accuracy tradeoff. Cognition’s Devin Desktop used a similar staged routing approach to achieve near-Opus 5 accuracy at a 28% lower average cost. The real story here is the architectural clarity: one model for planning, a much cheaper one for execution, and an intelligent switch between them. It’s a pragmatic admission that not every step in an agent’s chain of thought needs a PhD.