Cohere's new RTEB benchmark catches models faking their retrieval scores

Cohere just dropped a retrieval benchmark that’s designed to embarrass models that have been gaming the system. The Retrieval Embedding Benchmark — RTEB for short — takes direct aim at the dirty secret of embedding evaluation: when you train and test on the same public datasets, scores lie. Some models look brilliant on paper but fall apart when they encounter genuinely new data.

This isn’t speculation. The RTEB team already sees it happening. “A model with a significant performance drop between the open and the private datasets would suggest overfitting, providing a clear signal to the community,” they note in the announcement. And they’ve confirmed that the performance gap is already visible in certain models they’ve tested. That’s a polite way of saying some popular embeddings have been memorizing the answers.

The fix is a hybrid design. Half the benchmark uses fully public datasets — corpus, queries, relevance labels, the works — so anyone can reproduce results. The other half is sealed. Only MTEB maintainers can evaluate models against those private datasets, which span 20 languages and specific enterprise domains like healthcare, law, and finance. You get descriptive statistics and sample triplets, but never the full test. That’s how you measure generalization, not test-taking ability.

The choice of metric also matters. Rather than cobbling together a composite score, RTEB’s default leaderboard uses NDCG@10 — a retrieval-first metric that actually reflects what developers care about when building RAG pipelines, recommendation systems, or agent memory. Cohere is positioning this as the honest alternative to a benchmark ecosystem that’s quietly rewarded overfitting for years. The question now is whether the community will embrace a standard that makes their models look worse.