A tiny finetuned reranker just crushed public models 4x its size

You don’t need a massive, general-purpose reranker to get state-of-the-art search results. A new technical deep-dive and accompanying experiment shows that a focused finetuning run on your own domain can produce a model that punches far above its weight class.

The post walks through the entire training recipe using the Sentence Transformers library, covering datasets, loss functions, training arguments, evaluators, and the CrossEncoderTrainer. The practical result is startling. A compact model based on ModernBERT-base, which the author trained alongside the blog post, easily outperformed 13 of the most commonly used public reranker models on a held-out evaluation set. It even beat models four times larger.

Scaling the recipe up to a ModernBERT-large backbone produced a model that the author claims “blows all existing general-purpose reranker models out of the water” on this specific data. That framing is bold, but the underlying lesson is what matters: the “retrieve and rerank” two-stage search architecture is only as strong as its final filter. A general-purpose reranker has to be adequate for everyone, which prevents it from excelling for you.

Finetuning lets the cross-encoder learn the linguistic patterns and domain-specific signals that matter most in your data. The speed trade-off is well understood—cross-encoders process query-document pairs jointly, so they’re slower than bi-encoders at scale—but when you’re only rescoring the top-k candidates, raw throughput matters less than precision. This recipe makes a strong case that the returns from a targeted finetuning run are too large to ignore, especially when off-the-shelf models are leaving obvious relevance signals on the table.