Habana Labs’ second-generation Gaudi2 accelerator is finally getting real benchmark numbers, and they’re substantial. Hugging Face reran Philipp Schmid’s BERT pre-training script on the new hardware and watched the total training time collapse from 8 hours and 53 minutes to 2 hours and 55 minutes — a 3.04x speedup with zero code changes. That’s the headline, but the more interesting story is what happens when you actually use the hardware’s expanded memory.
Each Gaudi2 card packs 96GB of memory, triple the first-gen’s 32GB and a healthy margin over the A100 80GB. That headroom let the team push batch size from 32 to 64 samples per device, which accelerated convergence enough that 20k steps delivered the same loss as the original 65k. Total training time dropped to 1 hour and 33 minutes. Overall, that’s a 5.75x reduction in time-to-trained-model versus first-gen Gaudi, with throughput up 3.53x.
Against Nvidia’s A100 80GB, the numbers hold up too. Gaudi2 hit 1580.2 samples per second at batch size 32 versus the A100’s 981.6, and 1835.8 versus 1082.6 at batch size 64. That’s roughly the 1.8x speedup Habana claimed for BERT phase 1 pre-training, now confirmed independently. All runs were distributed across 8 devices — Gaudi in bf16/fp32 mixed precision, A100 in fp16.
The Stable Diffusion story is equally compelling. Optimum Habana 1.3 added SD support, and on Gaudi2 image generation latencies came in 3.51x faster than first-gen. One quirk: Gaudi generates images in batches rather than one at a time like GPUs, and the first two batches are slower due to model compilation overhead. Those were discarded from throughput calculations, which is standard practice but worth knowing before you benchmark your own workload.
Availability remains the friction point. The hardware is accessible through Intel Developer Cloud, but the setup process is a multi-step dance: register, request an instance, wait for validation, add a payment method and SSH key, accept terms, then finally launch. And if you terminate the instance, you start over from scratch. For teams evaluating whether to migrate from A100 infrastructure, the performance gap is now real enough to justify that hassle — but Intel still has work to do on developer onboarding.
