Hugging Face migrates 4.5 TB to Xet storage, turning 13-minute uploads into 0.1 seconds

Hugging Face just took its biggest swing yet at the file-size problem plaguing AI development. On February 20th, the Xet team migrated the first wave of Model and Dataset repositories off Git LFS and onto their custom Xet storage system, shifting 4.5 TB of data and roughly 6% of the Hub’s total download traffic in a single day. If you’ve ever waited 13 minutes to re-upload a 5 GB model because you tweaked a single line of metadata, this is the fix you didn’t know was coming.

The core of Xet isn’t just a storage swap—it’s a fundamental rethinking of how large files are handled. LFS, the old guard, deduplicates at the file level. Change one byte, and the whole multi-gigabyte file gets uploaded again. Xet uses content-defined chunking (CDC) to break files into roughly 64 KB chunks, then aggregates those into 64 MB blocks. Only the changed chunks ever travel over the wire. The Xet team tested this on an internal 5 GB SQLite database powering a metrics dashboard. Appending a routine 1 MB update took a tenth of a second with Xet, versus 13 minutes under LFS. That’s not an incremental improvement—it’s the difference between a workflow that feels broken and one that feels instant.

The migration itself was a carefully orchestrated event, complete with real-time observability via Grafana and Kibana dashboards that lit up as the team flipped repositories. The engineers even left a visible dip in the logs when they took a lunch break. Behind the scenes, a Xet-aware client handles the chunking and deduplication locally before upload, while a content-addressed store (CAS) enforces chunk-based deduplication on the server side. An LFS Bridge ensures backward compatibility for older, non-Xet clients, acting like a traditional LFS server and providing a single download URL.

Post-migration, things didn’t stay quiet. The team quickly spotted that CAS was downloading four times more data than it was returning to clients. The culprit? hf_transfer, a tool many users rely on to accelerate downloads, was firing off a flood of requests for 10 MB ranges within files. This kind of integration friction is the unglamorous reality of retrofitting a platform used by nearly two million developers. It’s a reminder that the hardest part of rebuilding core infrastructure isn’t the chunking algorithm—it’s making sure the thousand different ways people access the Hub don’t break. The migration proves the core tech works at scale, but the hf_transfer hiccup shows how much edge-case wrangling still lies ahead before Xet becomes the default for everyone.