Falcon-Edge proves 1.58-bit models can match 16-bit rivals—and you can fine-tune them

The Falcon-Edge series tackles the central headache of extreme model compression head-on: you usually have to pick between a tiny, fast model and one you can actually customize. This new collection of 1.58-bit language models—available in 1 billion and 3 billion parameter sizes, both as base and instruction-tuned variants—claims to give you both.

The secret sauce is a single pre-training run that spits out three usable formats: a standard bfloat16 model, a native BitNet ternary-weight model for blazing-fast inference, and a pre-quantized BitNet variant specifically designed for fine-tuning. That last part is a genuine unlock. Tinkering with aggressively quantized models has historically been a non-starter; Falcon-Edge’s architecture bakes in the hooks to make it practical. The team pre-trained on a 1.5 trillion token internal data mixture using a WSD learning rate scheduler.

Benchmarks on the old Hugging Face leaderboard v2 put these tiny models in a surprisingly respectable position. They perform on par with—and in some cases better than—similarly sized models that rely on heavier floating-point formats. Microsoft’s new BitNet model, a natural point of comparison, gets eclipsed by the instruction-tuned Falcon-Edge variants on the even older v1 leaderboard tasks. That’s not just a rounding error; it suggests the training recipe here is doing real work.

The cleverest trick might be how they derive the bfloat16 model. Instead of a separate training job, they exploit a mathematical quirk of BitNet’s activation quantization. Because the model was trained with fake 8-bit activation quantization, you can inject the weight scale after quantizing to approximate a full-precision version. End-to-end evaluations confirm the bfloat16 variant works as a drop-in replacement. This sidesteps the usual post-training quantization degradation, giving developers a practical on-ramp to experiment with 1.58-bit models without surrendering the ability to adapt them. The entire series is available now on Hugging Face.