Xet's chunking tech could slash Hugging Face storage by up to 100 TB

Hugging Face’s storage problem isn’t a secret. Model and dataset repositories have been ballooning for years, and the traditional approach—treating every file version as a wholly new object—is about as efficient as buying a new house every time you repaint a room. A team from Xet, now under the Hugging Face umbrella, is proposing a genuinely clever fix that’s already showing results.

Their method relies on content-defined chunking (CDC), which breaks files into variable-sized pieces based on the data itself rather than arbitrary byte counts. The magic is in a rolling hash that scans file contents and creates boundaries whenever a hash meets a specific condition. Identical chunks across versions? They’re only stored once. In one benchmark against Git LFS, Xet’s approach delivered a consistent 50% improvement in both storage and transfer performance. For the CORD-19 dataset—a collection of COVID-19 papers with 50 incremental updates—the difference in upload and download times was significant.

What could this mean for the Hub at large? The team’s deduplication estimator found that two versions of OpenAI’s GPT-2 model files overlapped substantially, with potential storage savings of 53%. Compression could push that to 63%. Across the entire platform, PyTorch model checkpoints alone account for roughly 200 TB of storage. At a 50% deduplication ratio, that’s an immediate 100 TB recovery, plus an ongoing reduction of 7-8 TB per month.

The real-world beneficiaries here are teams iterating on fine-tuned models and checkpoints, where changes between versions are often minimal. Fine-tuned models modify only a subset of parameters, making them prime candidates for deduplication ratios in the 30-85% range. The speed gains from only transferring modified chunks aren’t just nice to have—they cut waiting time for both humans and machines, which directly accelerates experimentation cycles.