The standard playbook for making LLMs efficient has been compress, quantize, then heal. But what happens when healing goes wrong because there’s no clean full-precision teacher left? That’s the gap a new paper from the GPT-OSS team tackles, and the result is a 4-bit model that outperforms the very checkpoint it was quantized from.
Quantization-Aware Healing (QAH) makes one key change to the usual distillation recipe: instead of distilling from the recovered bfloat16 checkpoint—which is itself a distilled approximation and therefore a degraded target—the method distills directly from the original, pre-compression model. Teacher and student don’t even share an architecture. The teacher is full-size and full-precision; the student is half the size and running in MXFP4. The team applied this to a GPT-OSS 120B model compressed to 60B parameters, and the 4-bit student beat the bfloat16 recovered checkpoint on 7 of 9 benchmarks.
The paper argues that the standard alternatives fall short in specific ways. Quantization-aware training (QAT) inserts fake-quantization operators and keeps fine-tuning on task loss, which requires re-running the entire post-training pipeline—SFT, RLHF, agentic tuning—through a noisier forward pass. It gets expensive and can become unstable if training overshoots. Quantization-aware distillation (QAD) avoids that cost but assumes a genuine full-precision version of the exact same architecture exists as a teacher. After structural compression, that assumption breaks.
There’s also a stability argument baked into the loss function itself. KL distillation ties the student to a fixed teacher distribution, so once the student catches up there’s no further pressure to drift. Cross-entropy task loss keeps pushing toward hard labels indefinitely. To handle 32k-token healing sequences within fixed GPU memory, the team reuses a chunked KL-divergence loss from a companion paper—computing the divergence one sequence slice at a time rather than materializing the full vocabulary-by-sequence grid.
The reframing is subtle but important: QAH treats quantization not as a lossy postprocessing step after healing, but as a second full distillation pass against the original teacher. The 4-bit student isn’t compensating for information lost to compression—it’s picking up information the earlier bfloat16 recovery never had the time or data to transfer. That inversion of the usual 4-bit-versus-16-bit relationship is the kind of result that makes efficiency teams sit up.
