MIT's Nunchaku tech puts 4-bit image generation inside Diffusers, no custom engine needed

Until now, running a diffusion model with 4-bit weights and activations meant juggling a separate inference library — more dependencies, more complexity. That changes with the integration of SVDQuant, the technique behind the Nunchaku engine, directly into Hugging Face’s Diffusers library. The new path, called Nunchaku Lite, lets you load a 4-bit quantized pipeline with a single from_pretrained() call. No local CUDA compilation. No custom pipeline classes. The necessary kernels download from the Hub automatically on first use.

The real story here is the quantization approach itself. Standard weight-only 4-bit methods shrink memory but often add latency because weights get dequantized on the fly. SVDQuant is different — it runs the transformer’s heavy attention and MLP layers with 4-bit weights and activations (W4A4). To stop large activation outliers from wrecking precision, it shifts them into the weights via a small 16-bit low-rank branch, then quantizes the remaining residual. The result is a genuine speedup, not just memory savings.

Nunchaku Lite doesn’t try to replicate the full Nunchaku engine’s speed. The original engine uses model-specific fused kernels — for example, fusing QKV projections or GELU/MLP paths — which are tightly bound to each architecture’s module layout. That deep integration is what gave the original its impressive performance, but it also made supporting new models a bespoke integration project. By patching Diffusers’ standard nn.Linear modules with generic SVDQuant and AWQ linear layers, Nunchaku Lite trades some peak performance for much broader compatibility. It’s a pragmatic move that still delivers around a 30% speedup over BF16 inference.

On an RTX 5090, a quantized ERNIE-Image-Turbo pipeline spits out a 1024x1024 image in 1.7 seconds using roughly 12 GB of VRAM — half the 24 GB the BF16 version needs. For Blackwell GPU owners, NVFP4 checkpoints are the default. Anyone on older hardware should use the INT4 variants. The companion diffuse-compressor toolkit also lets you quantize new architectures and publish them as standard Diffusers repos, which could quickly expand the library of plug-and-play 4-bit models. It’s not the screaming-fast, model-specific optimization of the original Nunchaku, but it’s a far more practical tool for the rest of us who just want things to work.