BLOOM hits under 1ms per token on 8 A100s, but setup is everything

Running the 176-billion-parameter BLOOM model isn’t just about having enough GPUs — it’s about choosing the right inference strategy. New benchmarks from Hugging Face reveal that Deepspeed-Inference with Tensor Parallelism and custom fused CUDA kernels can push throughput below 1 millisecond per token on an 8x80GB A100 node. That’s the kind of speed that makes real-time applications feasible for a model that demands 352GB of memory in bfloat16.

Accelerate, by contrast, takes a deliberately simple approach using naive Pipeline Parallelism. It hit roughly 1.2 milliseconds per token in the same tests. The trade-off is flexibility: Accelerate works out of the box with essentially any model, while Deepspeed-Inference’s fastest path often requires developer time to adapt custom kernels for models it hasn’t been tested on. The benchmarks used greedy generation of 100 tokens with previous token caching enabled, run on Jean Zay HPC’s 8xA100 configuration with 512GB of CPU memory and fast 3GB/s read speeds.

For teams with less hardware, int8 quantization via Deepspeed-Inference or BitsAndBytes cuts memory requirements in half, running on 4x80GB A100s at a modest throughput penalty. Deepspeed-ZeRO offers another path, processing multiple generation streams in parallel — effectively dividing throughput across 8 or 16 GPUs and enabling batch sizes up to 64 on the 8xA100 setup.

What’s striking here isn’t just the raw speed. It’s how quickly the loading time gap has closed. Deepspeed-Inference with pre-sharded weights gets you generating in about a minute, while Accelerate needs roughly two. Compare that to earlier approaches that could take ten minutes or more just to load checkpoints, and the operational picture changes: spinning up a BLOOM instance no longer feels like a heavyweight deployment decision. The remaining question is whether the custom kernel work required for peak Deepspeed performance is worth it for teams that aren’t running BLOOM specifically — or whether Accelerate’s plug-and-play simplicity wins out in practice.