Optimum-NVIDIA hits 28x faster LLM inference with a one-line change

Hugging Face and NVIDIA just made slow LLM inference feel like a choice rather than a constraint. The new Optimum-NVIDIA library, now available on Hugging Face, promises up to 28x better throughput and 1,200 tokens per second on NVIDIA hardware — and the API swap is almost embarrassingly simple. Change one import line from transformers to optimum.nvidia, flip a use_fp8=True flag, and you’re off. That’s the entire migration path for anyone already running a transformers pipeline.

The speed comes from two places. First, Optimum-NVIDIA is the first Hugging Face inference library to leverage FP8, the float8 format supported on NVIDIA’s Ada Lovelace and Hopper architectures. Second, it taps into TensorRT-LLM’s advanced compilation capabilities. The combination matters because FP8 lets you squeeze larger models onto a single GPU without the accuracy hit that typically accompanies aggressive quantization. The library ships with a predefined calibration strategy out of the box, though you can bring your own calibration dataset and tokenization if you’re picky about how the quantization behaves on your specific use case.

The performance numbers break down along two axes that matter for different reasons. First Token Latency — the time between hitting enter and seeing output start to stream — improves up to 3.3x over stock transformers. That’s the metric that makes a model feel snappy in interactive use. Throughput gets the headline-grabbing 28x figure, which is what you’ll care about when batching generations. Early tests on the H200 GPU show an additional 2x throughput boost over the H100 for LLaMA models, suggesting the gains scale with hardware generation.

Right now, support is limited to the LLaMAForCausalLM architecture, which covers LLaMA-based models including fine-tuned variants. The team says it’s expanding to other text generation architectures and plans to add In-Flight Batching and INT4 quantization. That INT4 piece is worth watching — it’s the same quantization level that made running 70B-parameter models on consumer hardware practical, and pairing it with TensorRT-LLM compilation could extend Optimum-NVIDIA’s appeal well beyond the current LLaMA-centric scope. For now, if you’re running LLaMA on NVIDIA GPUs and haven’t tried this, you’re leaving tokens on the table.