ZeroGPU cuts PyTorch cold starts by 1.8× with ahead-of-time compilation

The promise of ZeroGPU is simple and smart: stop paying for idle GPUs. Hugging Face Spaces grabs a GPU only when your model actually needs to run, then immediately releases it. This works beautifully for the spiky, unpredictable traffic most demos see. But that efficiency comes with a hidden cost. Since ZeroGPU spins up a fresh CUDA process for nearly every request, PyTorch’s default just-in-time compilation with torch.compile becomes a liability. The framework has to rebuild its optimized kernels from scratch each time — or crawl through a filesystem cache that can still take anywhere from 20 seconds to two minutes to restore. That’s a terrible user experience for anyone waiting on a demo.

The fix, as Hugging Face engineers detail in a new post, is ahead-of-time compilation. Using torch.export paired with AOTInductor, developers can compile a model once, export it, then reload the optimized version instantly inside any short-lived ZeroGPU process. The result is a 1.3× to 1.8× speedup on models like Flux, Wan, and LTX. The framework overhead that normally accompanies just-in-time compilation gets eliminated entirely.

The post walks through wiring up AoT compilation on ZeroGPU Spaces, starting with a standard Gradio app that loads black-forest-labs/FLUX.1-dev. It also explores more advanced territory — FP8 quantization to squeeze out extra performance and dynamic shapes for handling variable input sizes. Working demos are already live on the zerogpu-aoti organization for anyone who wants to kick the tires immediately.

There are caveats, of course. AoT compilation isn’t a universal drop-in. Export can fail on models with data-dependent control flow, and dynamic shapes require careful handling. But for the subset of models where it works cleanly, the payoff is real: demo responsiveness that finally matches the architecture’s promise. Hugging Face also notes that Pro users and Team/Enterprise org members can create ZeroGPU Spaces, while anyone can use them freely. Pro, Team, and Enterprise users get 8× the GPU quota. Additional MIG slice sizes, including a full 7g.141gb H200 profile, are slated for late 2025.