Sentence Transformers now ships a complete training pipeline for multi-vector, ColBERT-style retrieval models, and the results are blunt: a model finetuned in 14.5 hours on a single RTX 3090 outperformed every general-purpose retriever the author could find on a medical retrieval benchmark. Dense, sparse, lexical, multi-vector — none of them kept up.
The new MultiVectorEncoder in sentence-transformers handles late-interaction retrieval, where each token gets its own small vector and queries match documents token-by-token using MaxSim scoring. That preserves fine-grained signals that single-vector models flatten into one summary embedding. The tradeoff is a larger index, but the retrieval gains in specialized domains appear to justify it.
Why finetune at all? Most released models were built for short passages. ColBERT checkpoints truncate at 180 or 300 tokens, popular dense models at 256 or 512. On the author’s medical evaluation with passages averaging 941 tokens, truncation alone cost up to 0.24 NDCG@10 — more than any architectural difference between models. That’s a striking number: you can lose more performance to silent text truncation than to your choice of model family. LightOn hit the same wall with code retrieval and had to train LateOn-Code because general LateOn wasn’t cutting it.
The blogpost walks through every training component: the model itself, datasets, loss functions, training arguments, evaluators, and the trainer class that ties them together. Everything runs with a single pip install of sentence-transformers[train]. If your domain is medical, legal, financial, or your company’s internal documents, no official model is coming for you. The practical takeaway is that building a domain-specific multi-vector retriever is now a matter of hours on consumer hardware, not a research project requiring a cluster.
