The economics of training large language models are brutal. BERT cost roughly $7,000. GPT-3 reportedly ran $12 million. Those numbers shut out most researchers and hobbyists before they even start. But a team working on DeDLOC — Distributed Deep Learning in Open Collaborations — just demonstrated a different path: 40 volunteers with consumer GPUs pretrained sahajBERT, a Bengali language model, over ordinary internet connections.
The core insight is almost embarrassingly simple. Standard distributed training synchronizes gradients after every batch, which is fine on a fast local network but collapses when someone’s on a household data plan. DeDLOC flips the approach. Instead of frequent small syncs, it accumulates one massive batch across every volunteer device before taking a single optimizer step. If a peer disconnects — and volunteers will disconnect — their contribution just gets deducted from the accumulated batch. Other participants compensate. When new peers join, the target batch size arrives faster. No special recovery protocol needed.
That fault tolerance is the quiet breakthrough here. Most distributed systems treat node failure as an exception to handle. DeDLOC treats it as the default state of the network. The paper describes how different peers can take on different roles — some compute gradients, others aggregate, some just store parameters — which means a lab with a decent cluster and a gamer with an RTX 3080 can both participate meaningfully.
The results matter. sahajBERT achieved near state-of-the-art quality on Bengali downstream tasks, comparable to much larger models trained on hundreds of high-tier accelerators. That’s the part worth sitting with. A volunteer network produced something competitive with institutional infrastructure. It echoes what Folding@home did for protein simulation and Leela Chess Zero did for chess engines — but applied to the most expensive part of modern NLP.
The question now is scale. BERT is a few hundred million parameters. The volunteers trained it over weeks or months, not hours. Whether this approach can handle models an order of magnitude larger — or whether the volunteer pool has the patience — remains open. But for the long tail of low-resource languages, where no tech giant will ever fund a $12 million training run, DeDLOC might be the only realistic route to a competitive model.
