If you wanted to build a competitive large language model in 2022, the playbook was simple: throw more parameters at the problem. Bigger was better, full stop. BLOOM proved the open-source community could assemble a 176-billion-parameter model across 60 countries and 250 institutions, and Meta’s OPT family showed that even a company known for social media could ship a credible open model. The architecture was settled — decoder-only Transformers with minor tweaks — and the race was on to scale up.
But 2023 rewrote the rules. The biggest shift wasn’t a breakthrough in model architecture or a clever tokenization trick. It was the quiet realization that high-quality training data had become the scarcest resource in the field. The recipe itself hasn’t changed: you need an architecture, a dataset, a tokenizer, and hyperparameters. What changed is which ingredient you’re willing to fight over. Models now train on anywhere from several hundred billion to several trillion tokens, and the source article makes clear that curating those tokens — deciding what goes in and what stays out — has become the real competitive moat.
That’s a meaningful reversal from 2022, when the community obsessed over emergent abilities and scaling laws that suggested capability jumps at certain size thresholds. The open-source movement delivered on size: BLOOM remains the largest open multilingual model to date, trained on 350 billion tokens across 46 human languages and 13 programming languages, with most of its training data actually released to the public. But size alone stopped being the story. The conversation turned to what those models were actually fed — and whether the open community could keep up with the data pipelines of well-funded labs.
The underappreciated consequence is that fine-tuning became the great equalizer. Open pretrained models can be adapted for specific tasks at a fraction of the cost of training from scratch, which is exactly why researchers with limited compute budgets care so much about access to high-quality base models. The article frames this as a reason open models are valuable, but it also hints at a dependency: the open ecosystem increasingly relies on a handful of institutions willing to release both weights and enough information about their training data. If data is the new bottleneck, the organizations that control the best datasets — and choose whether to share them — will shape what open LLMs can do next. That’s not a scaling law. That’s a supply chain problem.
