Data quality remains the unglamorous workhorse of large language model training, and the BigCode project just offered a detailed look under the hood. The team behind the open-source code model has published a walkthrough of the MinHash and locality-sensitive hashing pipeline they use to strip near-duplicate documents from massive text corpora. It’s the same approach refined during the earlier BigScience effort, and the people involved say the payoff is concrete: less compute wasted, lower risk of benchmark contamination, and models that don’t parrot training data back verbatim.
The core idea is straightforward. Documents get chopped into word-level n-grams, or shingles, then each shingle is hashed multiple times to create a compact fingerprint. Documents that share enough fingerprint similarity get flagged as near-duplicates. The heavy lifting happens in the LSH step, where documents are grouped into bands so the system only compares likely matches instead of running every document against every other one. The article notes this process scales linearly with dataset size, which is what makes it feasible on corpora measured in terabytes.
One detail that jumps out: the authors link deduplication directly to training efficiency, citing prior work showing you can hit the same or better performance with fewer training steps after cleaning the data. For BigCode specifically, they confirmed the effect holds for code models, not just English-language LLMs. That matters because code datasets have their own duplication patterns — think forked repositories and copied Stack Overflow answers — which don’t always behave like natural language repeats.
What’s not fully settled is how aggressive to be. The article points to a comparison table of deduplication parameters across datasets, but the optimal Jaccard threshold and shingle size still depend on the corpus. Too loose and you leave contamination in; too strict and you risk cutting legitimate near-identical examples that help the model learn. The BigCode team seems to have landed on a middle ground, but they’re candid that parameter tuning is more art than settled science at this scale. For anyone building models on web-scraped text, the lesson is that deduplication isn’t a checkbox — it’s a design decision with real downstream consequences.
