Running Stable Diffusion XL out of the box is a bruising experience. The full-precision pipeline with default attention chews through 28GB of VRAM and takes 72.2 seconds to generate just four images on an A100. For anyone iterating on prompts — which is everyone, since diffusion outputs are a roll of the dice — that’s untenable. Hugging Face engineers ran a systematic benchmark to find out exactly how much speed and memory you can claw back with a handful of optimizations.
The first lever is precision. Swapping fp32 weights for fp16 cuts memory to 21.7GB and slashes inference time to 14.8 seconds. That’s nearly a full minute saved with a single torch_dtype argument. Modern GPUs have dedicated hardware for fp16 math, so the gains aren’t just from halving the bytes — the calculations themselves run faster. This alone transforms SDXL from a chore into something you can actually iterate with.
The second lever comes free if you’re on PyTorch 2.0 or newer. Scaled dot product attention, or SDPA, fuses Flash Attention, xFormers’ memory-efficient attention, and a native C++ implementation into one API. It’s enabled by default in Diffusers, which means the 11.4-second result requires no extra code beyond the fp16 pipeline. Attention memory scaling quadratically with sequence length has long been the silent killer of transformer-based image generation, so having optimized attention baked into the framework rather than bolted on as a third-party dependency is a meaningful shift.
From there, torch.compile with mode="reduce-overhead" pushes things further by JIT-compiling the UNet into optimized kernels. The article frames this as the next step after fp16 and SDPA, though the benchmark numbers for the compiled version aren’t spelled out in this excerpt. What’s clear is that the gap between a stock SDXL setup and an optimized one isn’t marginal — it’s the difference between a tool you use and a tool you wait on.
