NVIDIA dares the industry: reproduce our Nemotron Nano 3 scores yourself

Most model benchmarks are marketing theater. NVIDIA is betting that showing its work changes the game. Alongside the Nemotron 3 Nano 30B A3B release, the company published the complete evaluation recipe—prompts, configs, runtime settings, and logs—built on the open-source NeMo Evaluator library. The message is blunt: don’t just take our word for it, run the exact same pipeline on your own hardware and verify the results.

This is a direct challenge to a field where benchmark scores are often meaningless hype. NVIDIA argues that tiny, undisclosed tweaks to harness versions or inference settings can materially shift results, making model comparisons a fool’s errand. The NeMo Evaluator is designed to solve this by acting as a standardizing layer that sits on top of various testing harnesses. It coordinates benchmarks like LM Evaluation Harness and NeMo Skills under a single, consistent configuration, separating the evaluation logic from whatever inference backend you use.

For developers and researchers, the practical upshot is the end of the ‘one-off script’ era. The tool is built to scale from a quick single-benchmark sanity check to a full model card suite, capturing structured artifacts and logs for true auditability. You can inspect exactly how a score was calculated, which makes debugging and trust less of a black-box process.

Whether the community bites remains an open question. Publishing recipes is a power move, but it also invites the kind of scrutiny that can expose embarrassing flaws in a model’s reasoning that a single aggregate score conveniently hides. If other labs refuse to follow suit, it might not matter how transparent NVIDIA is—the benchmark environment stays poisoned. But if reproducibility becomes a competitive weapon, this could force a long-overdue cleanup of how AI progress is actually measured.