Embedding models have a scaling problem. As they get better at understanding text, they output ever-larger vectors—768 dimensions, 1024, more—which makes similarity search, classification, and recommendation systems slower and more expensive to run. A 2022 paper by Kusupati et al. proposed a fix inspired by Russian nesting dolls: train models so the most important information sits in the earliest dimensions, letting you truncate the rest without losing much performance.
The technique is called Matryoshka Representation Learning, and it’s now built into Sentence Transformers, one of the most widely used frameworks for training embedding models. During training, the loss function doesn’t just evaluate the full-size embedding—it evaluates truncated versions at multiple dimensions, say 768, 512, 256, 128, and 64. Those losses get summed together, so the optimizer learns to frontload critical information. The result is a single model that can emit useful embeddings at any of those sizes.
Why does this matter for practitioners? Two reasons stand out. First, shortlisting and reranking: you can run a fast, cheap nearest-neighbor search on 64-dimension embeddings to narrow a candidate pool, then rerank the survivors using full 768-dimension vectors. That’s a classic efficiency pattern that previously required separate models or dimensionality reduction steps. Second, the trade-off flexibility is real—storage costs, processing speed, and accuracy can now be dialed up or down without retraining anything.
The catch, as always, is that ‘without suffering too much on performance’ is doing a lot of work. Truncated embeddings will lose some fidelity, and the paper is careful to manage expectations on that front. But for production systems where vector databases are eating a growing share of infrastructure budgets, the ability to shrink embeddings on demand is a pragmatic win. The interactive demo on the original post is worth a look if you want to see the quality drop-off for yourself.
