DeepMind finds giant models beat tiny data in compute-optimal AI training

For years, the playbook was simple: if you had more compute, you trained bigger models on bigger datasets. New research from DeepMind suggests that approach has been wildly inefficient — and the optimal strategy flips conventional wisdom on its head.

The team, led by Jared Kaplan, studied empirical scaling laws for language model cross-entropy loss across more than seven orders of magnitude. Their findings are remarkably clean. Loss scales as a power-law with model size, dataset size, and compute — and within a broad range, architectural choices like width or depth barely move the needle. Simple equations, they found, govern both overfitting and training speed.

Here’s where it gets counterintuitive. Larger models turn out to be dramatically more sample-efficient than smaller ones. If you’re working with a fixed compute budget, you shouldn’t be training a moderately-sized model to convergence on a massive dataset. You should be training an enormous model on a relatively modest amount of data and stopping well before convergence. The paper’s equations let practitioners calculate exactly where that sweet spot sits.

This has real implications for how AI labs spend their money. Most current training runs, the authors imply, are compute-suboptimal — leaving performance on the table that could be captured by reallocating resources toward larger architectures trained for fewer steps. The question now is whether labs will actually overhaul their training pipelines based on these scaling laws, or keep doing what feels comfortable. Old habits die hard, even when the math says otherwise.