Quanto FP8 Quantization Cuts SD3 Memory Use by 33% on a Single GPU

Running Stable Diffusion 3 in FP16 eats 18.765 GB of GPU memory. That’s a problem for anyone without a datacenter card. Hugging Face’s Quanto toolkit, built on PyTorch and bundled with Optimum, might be the fix diffusion users have been sleeping on.

Quantization is old news in the LLM world, but diffusion pipelines have lagged behind. The Hugging Face team benchmarked three Transformer-based text-to-image models — PixArt-Sigma, Stable Diffusion 3, and Aura Flow — on an H100 with PyTorch 2.4.0. The approach is simple: call quantize() on the pipeline’s transformer with FP8 weights, then freeze() to swap in the quantized parameters. That alone delivers meaningful savings with what they describe as almost no quality degradation and only slightly higher latency.

The bigger win comes from quantizing the text encoder too. Since modern pipelines stack multiple encoders — SD3 uses three — the combined savings add up fast. SD3 presents a wrinkle: quantizing the second text encoder doesn’t work well, so the team recommends quantizing either the first encoder, the third, or both, while leaving the middle one alone. The diffusion transformer gets quantized in all cases.

Some other findings worth noting: bfloat16 runs faster than FP16 on H100 and 4090-class hardware. INT8 quantization beats FP8 on inference latency, especially when you fuse the attention QKV projections to give the int8 kernels thicker dimensions to chew on. The team also poked at INT4 with bfloat16, though details there are still limited. For developers stuck on consumer GPUs, these techniques lower the barrier to experimenting with models that would otherwise be out of reach.