The BigCode collaboration — Hugging Face, ServiceNow, and NVIDIA — has shipped StarCoder2, a family of open code LLMs in 3B, 7B, and 15B parameter sizes. The flagship 15B model trained on 4+ trillion tokens across 600+ programming languages and, per the team, outperforms everything in its size class while matching models twice as large on many evaluations. That’s a meaningful bar: a 15B parameter model trading blows with 33B+ competitors signals that training data quality and the Fill-in-the-Middle objective are doing heavy lifting, not raw parameter count.
The smaller models aren’t afterthoughts. ServiceNow trained the 3B on 3+ trillion tokens across 17 languages, and Hugging Face handled the 7B on 3.5+ trillion tokens. The 3B model matches the performance of StarCoder1-15B — a striking generational leap, given the size difference. All three models share the same architecture: Grouped Query Attention, a 16,384-token context window with 4,096-token sliding window attention, and Fill-in-the-Middle training, which lets the models complete code with surrounding context rather than just left-to-right.
The models are trained on The Stack v2, a code dataset derived from the Software Heritage archive — the UNESCO-backed initiative to preserve all publicly available source code. Version 2 improves language detection, license filtering, and heuristics over the original, and groups examples by repository, which enables training with repository-level context. That’s a notable shift: most code datasets flatten files into isolated snippets, destroying the cross-file dependencies developers actually rely on. Repository-grouped training could be why the models perform so well on real-world code understanding tasks.
Licensing and governance are unusually polished for an open release. The models use the BigCode OpenRAIL-M v1 license, and the team provides a full-text search tool and a membership test for the pretraining dataset — practical tooling for developers who need to check whether their code was included. Between the three size classes, the VSCode extension, and the permissively licensed weights, BigCode is making a serious play for the open-source code assistant market that Code Llama and DeepSeek Coder currently contest. Whether StarCoder2’s performance holds up outside curated benchmarks is the next question — but the initial numbers are hard to dismiss.
