Hugging Face just kicked off a new benchmarking series focused on squeezing real performance out of BERT-like models on CPUs, and the initial results are a reality check for anyone assuming more hardware equals faster inference. The first post covers the hardware side, running everything on an AWS c5.metal instance with an Intel Xeon Platinum 8275 CPU — 48 cores, 96 threads, AVX512, and Intel’s VNNI instructions for quantized workloads.
The team rebuilt their benchmarking framework from scratch using Facebook’s Hydra configuration library, which means every run now tracks exactly what was changed. That’s a big deal for reproducibility, since the old approach made it nearly impossible to compare results across different setups. The new framework supports PyTorch, TensorFlow, TorchScript, XLA, and ONNX Runtime — the last one being particularly interesting because ONNX Runtime ships optimizations specifically targeting transformer models.
What’s notably absent from this first post is any discussion of distillation or pruning. Hugging Face says those are coming later, along with deeper dives into quantization using float16, int8, and int4 representations. The baseline numbers themselves show that out-of-the-box performance leaves a lot on the table, especially around core scaling. Throwing more cores at a single inference task doesn’t linearly improve throughput — a pattern familiar to anyone who’s watched database benchmarks hit memory bandwidth walls.
That’s the Inblix angle here: the gap between theoretical CPU capability and what default PyTorch or TensorFlow configs actually deliver is still wide. If you’re deploying BERT in production without touching thread counts, batch sizing, or runtime selection, you’re likely paying for hardware you never use. The series promises to map out exactly which knobs matter, and based on the c5.metal setup, NUMA topology is clearly going to be part of that conversation.
