Llama 8B shrank to 1.58 bits per parameter and still beat a full-size model

Microsoft Research’s BitNet architecture promised a world where LLM parameters are just -1, 0, or 1, slashing memory and energy. But that required training a model from scratch, a non-starter for most teams. A new project just cracked the code on retrofitting existing models.

Engineers successfully fine-tuned a Llama 3 8B model into the 1.58-bit BitNet format and released three variants under the HF1BitLLM organization on Hugging Face. Two were trained on 10 billion tokens with different setups, and a third saw a full 100 billion tokens. The punchline? These quantized 8B models outperformed the original Llama 1 7B on the MMLU benchmark. That’s a 1.58-bit model, using mostly INT8 addition for matrix math, outscoring a full-precision predecessor.

The energy savings are staggering. The team cites a theoretical 71.4x reduction in arithmetic operations energy for matrix multiplication compared to a standard Llama baseline using FP16. Instead of heavy multiply-accumulate operations, BitNet’s BitLinear layers rely almost entirely on addition. For anyone running inference at scale, this flips the cost equation on its head.

Under the hood, the magic is in a training trick called the Straight Through Estimator. Ternary weights (-1, 0, 1) are non-differentiable, which would normally break backpropagation. The STE lets gradients pass through the rounding step as if it didn’t exist, enabling standard optimization. During training, weights are quantized on the fly using symmetric per-tensor scaling, while activations are squeezed to 8-bit precision with absmax per-token quantization. The team even landed a pull request to bake “bitnet” as a native quantization method directly into Hugging Face Transformers, meaning you can load these models with zero API changes. Given how brutal the economics of large-scale inference have become, a drop-in path to 1.58-bit fine-tuning isn’t just clever—it’s a pragmatic escape hatch.