For years, the classical statistical view held that overly complex models were a recipe for disaster — overfitting would tank your test performance. Then the deep learning revolution arrived and flipped that script: bigger models, counterintuitively, almost always worked better. What a team of researchers from OpenAI has now shown is that both camps were simultaneously right and wrong, and the truth is far stranger than a simple rule of thumb.
In a new study examining CNNs, ResNets, and transformers, the team demonstrates that model performance undergoes a ‘double descent’ — test error improves, then gets worse, then improves again as you add parameters, training time, or even more data. The worst performance predictably clusters around a ‘critical regime’ where the model is just barely big enough to memorize its training set. That interpolation threshold is where things fall apart; the model has exactly one way to fit the noisy labels and no flexibility to find a solution that generalizes well.
The implications are genuinely unsettling. There’s a regime where training on 4.5x more data actually hurts performance, as observed in transformer-based translation tasks. The extra samples shift the critical threshold to the right, so a model that was previously in a sweet spot suddenly finds itself stranded at the peak of the error curve. Adding label noise amplifies the entire phenomenon, making the peak far more pronounced — the paper notes that without this noise, the effect can be subtle enough to miss entirely.
This isn’t just about model size, either. The team documents an ‘epoch-wise double descent’ where, for a fixed architecture, test error decreases, increases, and then decreases again over the course of training. As the authors put it, at the interpolation threshold there are effectively no good models that both fit the train set and perform well — but once you push past it into over-parameterized territory, many such models exist and SGD seems to find them for reasons no one fully understands yet. That last admission, that the implicit bias driving this is still a mystery, is what makes this paper as much a call for further research as a definitive explanation.
