Tom Aarsen's finetuned 2B model beats giants 4x its size on visual doc search

General-purpose models are jacks of all trades, masters of none. Tom Aarsen just proved exactly how much that generality costs you. Starting with the Qwen/Qwen3-VL-Embedding-2B model, he finetuned it for a single, unforgiving task: Visual Document Retrieval. That’s the art of matching a text query like “What was Q3 revenue?” to the correct screenshot of a document, complete with charts, tables, and layout quirks. The results weren’t incremental. They were a rout.

The finetuned model, now available as tomaarsen/Qwen3-VL-Embedding-2B-vdr, hit an NDCG@10 score of 0.947 on his evaluation data. The base model it started from? A respectable 0.888. But the truly stinging detail is that this 2-billion-parameter model then proceeded to outperform every existing VDR model Aarsen tested against. We’re talking models up to four times its size. It’s a textbook case of why off-the-shelf solutions, however impressive, often leave massive performance on the table for specialized work.

The magic isn’t in some radical new architecture—it’s in the data and the standard Sentence Transformers training pipeline. Aarsen walks through the entire process, showing how you can load a model from a VLM checkpoint like Qwen3-VL-2B even if it’s never been used for embeddings, with the library automatically handling modality detection and processor setup. He points out that the key difference from text-only training is simply that your dataset now includes images, and the model’s processor handles the rest automatically.

What strikes me here isn’t just the score, but the accessibility. The same SentenceTransformerTrainer that developers use for text models works for this. You tweak processor_kwargs to control image resolution and memory, pass in your multimodal dataset, and go. The barrier to building a specialized retrieval system that crushes a general-purpose one isn’t high. It’s just a question of whether you’re willing to gather the data. Aarsen’s experiment is a blunt reminder that the best model for your specific problem is almost certainly not the one topping a generic leaderboard.