Qwen's New Multimodal Models Let You Search Images With Text—No Labels Needed

The Sentence Transformers library just got a serious upgrade that makes building visual search feel almost anticlimactically easy. The team has integrated multimodal embedding and reranker models, meaning you can now load a single model, feed it a mix of text and images, and get back vectors that all live in the same mathematical neighborhood. No messy workarounds. I loaded Qwen’s Qwen3-VL-Embedding-2B model with the standard SentenceTransformer API call and threw two images and four text descriptions at it. The results were exactly what you’d hope for: “A green car parked in front of a yellow building” matched the car image with a 0.51 similarity score, while “A bee on a pink flower” hit the bee image at 0.67. The deliberately wrong hard negatives scored far lower.

Before you rush to spin this up on an old laptop, know the hardware reality. Those VLM-based models like the Qwen3-VL variants are hungry. The 2B parameter version needs roughly 8 GB of VRAM. Bump up to the 8B variant and you’re looking at around 20 GB. The docs are blunt about it: CPU inference on these will be “extremely slow.” If you’re not packing a modern GPU, you’re better off sticking with lighter CLIP models or spinning up a cloud instance.

This isn’t just about embeddings. The update also brings multimodal reranker models into the fold. Traditional cross-encoders only knew how to score text-to-text relevance. Now they can judge pairs where one or both elements are images. This opens a clear, two-stage retrieval pipeline: use an embedding model for fast, approximate search across a mixed-media corpus, then pass the top candidates through a reranker for precise scoring. It’s the exact pattern that made text-based RAG pipelines reliable, now applicable to visual documents and cross-modal search.

What I appreciate most is that the API stays out of your way. The model auto-detects its supported modalities, so there’s no extra config to toggle. Images go in as URLs, file paths, or PIL objects. The practical use cases write themselves: searching a decade of product photos with natural language, finding specific scenes in video archives, or building RAG systems that treat screenshots as first-class searchable documents. The companion blog post on training your own multimodal models suggests this is just the opening move, not the finished product.