Habana Labs’ second-generation Gaudi2 accelerator is delivering inference on BLOOMZ, the 176-billion-parameter open-source language model from BigScience, at 1.42 times the speed of Nvidia’s A100 80GB. The benchmark, run with DeepSpeed-inference in 16-bit precision across 8 accelerator devices, used a batch of one sample and greedy generation of 100 output tokens. That’s a meaningful gap for anyone serving massive language models in production.
The performance spread gets wider with smaller checkpoints. On BLOOMZ-7B, Gaudi2 outpaced the A100 by 2.89x. The A100 actually performed better on a single device for the smaller model, while Gaudi2 continued to benefit from model parallelism across its eight HPUs. Each HPU packs 96GB of memory, so a single server provides 768GB total—enough headroom for a model that needs 352GB just to hold its weights in 16-bit precision.
Habana’s SynapseAI SDK now supports HPU graphs and DeepSpeed-inference, two features that matter for latency-sensitive applications. CUDA graphs, by contrast, still aren’t compatible with model parallelism in DeepSpeed as of version 0.8.2. The timing here is notable: SynapseAI 1.8 introduced DeepSpeed-inference support, and by the Optimum Habana 1.6 and SynapseAI 1.10 releases, the Gaudi2 numbers had jumped by that 1.42x factor.
First-generation Gaudi hardware remains a budget option, with AWS DL1 instances running about $13 per hour. The latency is higher, but for teams experimenting with BLOOMZ or running batch workloads, the economics may still pencil out. The broader signal is that Nvidia’s dominance in AI inference is no longer a given—specialized silicon from Intel-owned Habana is now competitive on one of the largest open models available.
