A 0.6B model just beat SAM 3 on grounding—here's the simple trick that made it work

The team behind Falcon Perception asked a question most of the field has been dodging: why are vision systems still stitched-together pipelines? Their answer is a 0.6-billion-parameter Transformer that ditches the modular approach entirely. Instead of a frozen vision backbone feeding a separate language module, everything—image patches, text, and task tokens—gets dumped into one sequence from the first layer. The result hits a 68.0 Macro-F1 on the SA-Co benchmark, a solid jump over SAM 3’s 62.3.

The secret isn’t just smashing everything together. It’s a hybrid attention mask that lets the model cheat in a smart way. Image tokens get to look at all other image tokens bidirectionally, building a clean global picture like a standard vision encoder. But the text and task tokens that actually do the predicting are forced to look forward causally. It’s a single backbone cosplaying as two different architectures, and it works. To handle the messy reality of predicting a variable number of objects, they cooked up a “Chain-of-Perception” sequence: a coordinate token first to lock onto the object, then a size token to define its extent, and finally a single segmentation embedding that produces a full mask via a dot product with upsampled features. Committing to geometry before the mask resolves the ambiguity that kills most dense predictors.

The main flaw they’re honest about is presence calibration—the model’s ability to know when no object is there. Its MCC score of 0.64 lags behind SAM 3’s 0.82. That’s the next hill to climb. They also released PBench, a diagnostic benchmark that doesn’t just give a single number but breaks performance down by specific capabilities like OCR-guided disambiguation and handling dense crowds. It’s the kind of tool that tells you exactly why your model is failing, not just that it is.

Alongside this, they dropped Falcon OCR, a tiny 0.3B model that scores 88.6 on OmniDocBench while apparently having higher throughput than any open-source alternative. The dual release feels less like a product launch and more like a proof of concept: that small, single-backbone models can punch far above their weight class if you nail the attention pattern and output interface. The question now is whether this approach scales or if it’s a party trick for sub-billion-parameter models.