Training a 176-billion-parameter language model isn’t something a scrappy team of volunteers typically pulls off. But that’s exactly what happened with BLOOM, the multilingual open-source model that emerged from a collaboration led by Hugging Face’s Thomas Wolf. The project ran from March to July 2022, consuming roughly one million compute hours across 384 NVIDIA A100 80GB GPUs on France’s Jean Zay supercomputer.
The hardware footprint alone tells you how serious this was. Jean Zay, managed by GENCI and installed at IDRIS, contributed the compute through a formal grant. Each of the 48 nodes packed eight A100s connected via NVLink 4, with AMD EPYC 7543 CPUs and 512GB of system memory per node. Full checkpoints with fp32 optimizer states and bf16 plus fp32 weights hit a staggering 2.3TB each — just the bf16 weights alone are 329GB.
What made BLOOM technically distinctive was the software stack. The team trained on Megatron-DeepSpeed, a fork combining DeepSpeed’s ZeRO sharding and pipeline parallelism with Megatron-LM’s tensor parallelism from NVIDIA. That 3D parallelism approach is what made training a model this size feasible outside a hyperscaler’s walls. DeepSpeed’s developers, including Olatunji Ruwase and Jeff Rasley, spent weeks working directly on the project’s needs, while NVIDIA’s Megatron-LM team fielded questions and offered what the article calls ‘first class experiential advice.’
The dataset was equally ambitious: 1.5TB of deduplicated, cleaned text spanning 46 languages, converted into 350 billion unique tokens with a vocabulary of 250,680 tokens. That multilingual focus sets BLOOM apart from GPT-3, which skews heavily English. And unlike the walled-garden approach at OpenAI or Google, the entire result was made openly accessible. That’s the real story here — a community-funded, volunteer-heavy effort proving that cutting-edge model training doesn’t have to be locked inside a corporate lab.
