The team behind Cosmopedia just released what they’re calling the largest open synthetic dataset ever built: 30 million files spanning 25 billion tokens, all generated by Mixtral-8x7B-Instruct-v0.1. The motivation is straightforward. Microsoft’s Phi models demonstrated that pre-training on synthetic textbooks can outperform larger models trained on raw web data, but the company never released its datasets or full methodology. Phi-2 alone saw over 617,000 downloads last month and sits among the top 20 most-liked models on Hugging Face, so the appetite for replication is real.
What makes this release different is that it’s fully reproducible. The pipeline code, the dataset, and a 1B model called cosmo-1b trained on it are all public. The technical approach combines two seed data strategies: conditioning on curated sources like textbooks and encyclopedia entries, and conditioning on web samples for diversity. The result, according to the team, is less than 1% duplicate content across the corpus.
The honest detail here is where the real work went. Rather than tales of orchestrating hundreds of H100s, the team admits most of their time was spent on meticulous prompt engineering. Generating synthetic data at scale isn’t the hard part — maintaining diversity is. You can burn compute generating billions of samples, but if they all resemble each other, you’ve wasted the GPUs. That’s why prompt curation became the bottleneck.
Skeptics have called synthetic pre-training “garbage in, garbage out,” and the benchmark overfitting critique against Phi isn’t going away. But the counterargument writes itself: if a 1B model trained on this data performs competitively, the open community finally has a foundation to test those claims directly instead of arguing about them in the abstract.
