Uploading a 200GB model to Hugging Face shouldn’t feel like a lunch break you’ll never get back. The Xet team is now rolling out a production-grade content-defined chunking (CDC) system that, in some cases, makes transfers 2-3 times faster. They’ve open-sourced the core of it—xet-core and hf_xet, a Rust-powered integration for huggingface_hub—which ditches the old file-centric approach for one that slices data into chunks and reassembles them intelligently.
But here’s where the engineering gets genuinely interesting: pure chunking is a database nightmare. At Hugging Face’s scale—nearly 45PB across 2 million repositories—a naive approach would create roughly 690 billion entries in the content-addressed store. That volume would crush network requests, bloat metadata databases, and send infrastructure bills to the moon. The team’s key insight is that deduplication isn’t the goal; it’s just a tool to improve the builder experience. So they loosened the constraint.
Their fix is aggregation. Instead of tracking individual chunks, the system bundles them into content-addressed blocks up to 64MB, slashing CAS entries by a factor of 1,000. It also generates “shards”—mappings between files and chunks that reference those blocks—so the system knows exactly which parts of a file changed without re-uploading everything. The real cleverness, though, is in “key chunks.” This is a 0.1% subset of all chunks, selected via a simple modulo condition on the hash. A global index over these key chunks and their shards lets the client query a tiny sample and then leverage spatial locality to deduce that nearby chunks likely already exist. No per-chunk network call required.
The 3.5PB of .gguf files already on the Hub—mostly quantized versions of other models—are a particularly juicy target. Quantization naturally restricts value ranges, which creates the kind of repetition that makes deduplication shine. This isn’t just an academic cleanup; it directly accelerates the iteration cycle for teams shipping model variants. If you’ve ever stared at a progress bar while pushing a fine-tune, the implication is clear: less waiting, more shipping.
