Scaling Laws
Empirical relationships showing that model performance improves predictably as you increase parameters, training data, and compute — and that the three must be scaled together to avoid waste.
Scaling laws describe how the loss of a language model falls as training compute, dataset size, and parameter count grow. First documented systematically in the GPT-2 and GPT-3 era, they gave labs a rare thing: the ability to forecast model quality before spending tens of millions of dollars on a training run.
The key practical insight is balance. If you scale parameters but not data, the model overfits or under-trains; if you scale data but not parameters, you saturate. The “Chinchilla” results refined the recipe, showing that many early large models were under-trained relative to their size and that more tokens for a given compute budget usually wins.
Scaling laws have limits and caveats:
- They predict loss, not usefulness. Benchmarks and real-world capability can improve unevenly.
- Data is finite. High-quality text is being exhausted, pushing focus toward synthetic data, multimodality, and better data curation.
- Inference cost grows with use, so labs increasingly trade training compute for cheaper serving.
Whether smooth scaling continues indefinitely, or hits diminishing returns, remains one of the defining debates in AI research.