Intel’s 4th generation Xeon processors are not the first place most developers look for running 15-billion-parameter code models. But new optimization work from Hugging Face and Intel suggests that assumption needs revisiting. Using weight-only and static quantization combined with assisted generation, the teams report more than 7x inference acceleration for BigCode’s StarCoder-15B on Xeon hardware.
The bottleneck, as the researchers explain, is memory bandwidth. Every token generated in an auto-regressive model requires loading the entire model from DRAM to the CPU. Quantization attacks this directly by shrinking the model footprint. The team applied SmoothQuant, a post-training quantization algorithm designed specifically for LLMs, to produce Q8-StarCoder. Standard static quantization struggles with LLMs because activations contain large-magnitude outliers in specific channels. SmoothQuant addresses this by applying smoothing scaling factors to both activations and weights before quantization, evening out those outliers.
Testing on the HumanEval benchmark showed Q8-StarCoder holds no accuracy loss versus the baseline—in fact, the team notes a slight improvement. That’s a meaningful result, because the fear with aggressive quantization is always that code generation quality will degrade. It didn’t here.
The work builds on Intel’s Advanced Matrix Extensions (AMX), the built-in BF16 and INT8 GEMM accelerators present in every core of 4th gen Xeon chips. Those accelerators, exposed through PyTorch 2.0 and Intel Extension for PyTorch, are what make this kind of inference feasible on commodity server hardware rather than requiring specialized GPUs. For teams running code completion models in production, that’s a significant cost consideration—and it puts pressure on the assumption that GPU inference is always the right answer for LLM serving.
