Image segmentation used to be three separate problems. Instance segmentation picked out individual objects. Semantic segmentation labeled every pixel. Panoptic segmentation did both. Each required its own architecture — until DETR arrived in 2020 and showed a Transformer decoder could handle all three with the same “binary mask classification” approach. MaskFormer refined the idea. Mask2Former pushed it further still.
Now OneFormer takes the next logical step. Instead of training separate models for each task, it trains once on a panoptic dataset and conditions the model with a text encoder — feed it “instance,” “semantic,” or “panoptic” and it adjusts accordingly. The result: state-of-the-art performance across all three segmentation tasks from a single checkpoint. That’s genuinely useful, because it means teams no longer need to maintain three separate model pipelines, each with its own training runs, evaluation metrics, and deployment quirks.
The tradeoff is latency. OneFormer adds a text encoder on top of the Mask2Former-style architecture, which makes it more accurate but slower. For real-time applications like autonomous driving or video processing, that extra overhead might push developers back toward Mask2Former. For offline batch work — medical imaging, satellite analysis, cataloging — the accuracy bump likely justifies the cost. The paper’s authors released 30 checkpoints for Mask2Former alone, trained on various datasets, so the ecosystem around these universal architectures is maturing quickly.
Under the hood, both models follow the same blueprint: a backbone (ResNet, Swin Transformer, or the newer DiNAT) extracts low-resolution features, a pixel decoder upsamples them, and a Transformer decoder turns a set of queries into binary masks and class predictions. The shift away from per-pixel classification is now essentially complete — universal segmentation architectures have won. What’s still open is whether the text-conditioning trick in OneFormer becomes the default pattern, or whether it’s a stepping stone toward models that understand segmentation instructions in plain language.
