Hugging Face transformers now runs at native vLLM speed with zero porting

The wall between using a model in Hugging Face transformers and deploying it at top speed in vLLM just crumbled. A new upgrade to the transformers modeling backend means you can now serve compatible models inside vLLM and get performance that matches hand-written, native vLLM implementations. No custom port required. You flip one flag: —model-impl transformers.

The team put this to the test across three very different Qwen3 configurations: a 4B dense model on a single GPU, a 32B dense model split across two GPUs with tensor parallelism, and a monster 235B-parameter FP8 Mixture-of-Experts model running data and expert parallelism across 8 H100 GPUs. In every case, the updated transformers backend matched the speed of vLLM’s own meticulously tuned code.

Under the hood, the upgrade gets clever. Instead of just swapping in vLLM’s attention kernels at runtime—which was the old approach—the system now uses torch.fx to statically analyze the model’s computation graph. It hunts for patterns that can be optimized, then uses Python’s ast module to literally rewrite parts of the source code on the fly. This lets it fuse operations into vLLM’s ultra-optimized kernels, like those used for Expert Parallelization in MoE models, and automatically infer parallelization plans for tensor and pipeline parallelism.

One detail that should make researchers happy: the manipulated models remain fully compilable with torch.compile and CUDA Graphs, and unlike dedicated vLLM implementations, transformers model code can still be used for training, evals, and RL rollouts. Models using linear attention aren’t supported yet, and custom repos that don’t follow the standard conventions will likely break. But for the 450+ architectures already in transformers, this removes a major friction point between experimentation and production.