Ettin benchmarks prove encoder models crush decoders at 4X the efficiency

For years, the industry has guessed whether bidirectional encoders or causal decoders were better for non-generative tasks, but every comparison was apples-to-oranges. Different data, different training tricks, different budgets. The team behind Ettin finally ran the controlled experiment the field needed.

They took the ModernBERT recipe—a proven, modernized encoder formula—and applied it identically to both encoder-only and decoder-only models, ranging from 17M to 1B parameters. The result is a suite of six models, all trained on 2 trillion tokens of fully open data with the exact same architecture shapes. The only variables? Bidirectional vs. causal attention, and masked language modeling vs. causal language modeling objectives. That’s it.

The benchmark scorecard is striking. On the Massive Text Embedding Benchmark (MTEB), Ettin’s 1B encoder scores 63.3 while the matching decoder limps to 51.1. That’s not a gap—it’s a chasm. The trend holds across sizes. A 17M encoder roughly matches a 110M decoder, suggesting encoders can deliver similar performance at about one-quarter the parameter count for classification and retrieval tasks. Given that encoders are already faster and more memory-efficient at inference, this efficiency multiplier is a big deal for production systems that need to minimize cost and latency.

The decoder models aren’t slouches in absolute terms. They beat Llama 3.2 1B and SmolLM2 on knowledge-intensive tasks like SciQ, proving the training recipe itself is genuinely strong. But in a head-to-head fight on discriminative benchmarks, the encoder’s bidirectional attention provides a fundamental advantage that no amount of scale or training wizardry seems to overcome. The paper also probes whether you can convert a trained encoder into a competitive decoder and vice versa—early results suggest that while some cross-architecture transfer is possible, the performance tax is significant.

For engineering teams, the implication is clear. If you are not generating text and your workload involves search, classification, or clustering, a decoder model is likely the wrong tool. The efficiency gains from switching to an encoder are no longer theoretical—they are measured and reproducible.