The visual document retrieval space has been coasting on repurposed generative models for a while now. You take a pretrained vision tower, bolt it onto a causal language model, and hope the projector does enough work. NeoMME, a new family of multilingual multimodal encoders from the folks behind ColPali-style retrieval, throws that playbook out. The 260M and 800M parameter models use a single bidirectional Transformer to process both text tokens and raw image patches directly. No separate SigLIP2 vision tower. No causal decoder. Everything trained from scratch with a masked discrete-diffusion objective.
The architecture choices matter. Images get split into non-overlapping 32×32 patches and projected through a small MLP, then enter the same encoder stack as text. Dynamic resolution means a dense 4K document page gets more tokens than a small screenshot, and the 16,384-token context handles up to two standard 4K images. The training recipe is clever too: multimodal examples use corruption rates between 0.3 and 1, forcing the model to ground masked text in image content rather than guessing from surrounding words. Each model chews through about 524 billion packed tokens, with NorMuon optimizer compensating for a text budget that’s a quarter of ModernBERT’s 2 trillion.
When fine-tuned for retrieval using ColPali’s page-image approach, NeoMME-Retriever lands on the ViDoRe v3 Pareto frontier for nDCG@10 versus model size. The 260M variant encodes roughly 51 pages per second on an L40S at 2048×2048 resolution—about twice ColModernVBERT’s throughput. But the real headline is storage. Hierarchical token pooling plus asymmetric quantization shrinks late-interaction index storage from about 1.5 MB to 6 kB per page. That’s a 255x reduction while keeping more than 95% of baseline nDCG@10.
This is the kind of efficiency gain that changes what’s practical to deploy. A million-page corpus goes from needing 1.5 terabytes of index storage to about 6 gigabytes—a number that fits comfortably on a single commodity SSD instead of a rack of NVMe drives. The retrieval community has been pushing hard on late-interaction quality since ColPali showed page-image embeddings could beat text-only baselines, but storage costs remained the awkward asterisk. NeoMME attacks that directly. Checkpoints are on Hugging Face Transformers under Apache 2.0, which should make this an easy drop-in for anyone already using the ColPali stack.
