Hugging Face silently moves 500,000+ repos off Git LFS with near-zero complaints

Hugging Face has pulled off what might be the quietest massive infrastructure migration in developer tools, transitioning over 500,000 repositories from Git LFS to the Xet storage system while most users never noticed. More than a million Hub users are now on Xet, which became the default for new accounts and organizations back in May. The real story isn’t just the scale — it’s the engineering that made the switch invisible.

The linchpin is something the team calls the Git LFS Bridge. When a client that doesn’t understand Xet’s chunk-based protocol requests a file, the bridge steps in, reconstructs the data from S3 on the fly, and serves it through a single presigned URL that mimics the old LFS behavior. In other words, older versions of huggingface_hub or huggingface.js just keep working. The bridge lets Hugging Face maintain backward compatibility without forcing every user to update their toolchain immediately — a design choice that probably prevented thousands of support tickets.

Behind the scenes, a background migration system runs around the clock, chewing through petabytes of models and datasets. A webhook triggers an orchestrator that batches files into jobs — capping at either 1,000 files or 500MB — and hands them off to worker pods. Those pods download the LFS files, then re-upload them to the Xet content-addressed store using xet-core. It’s a continuous sync operation that keeps everything consistent while the old and new systems coexist.

When the team stress-tested this with bartowski’s nearly 500 terabytes across 2,000 repositories, the cracks showed fast. Temporary shard files for global deduplication filled disks because they were written to a different mount point than the cache and never cleaned up. The CAS infrastructure, originally scaled for bursty downloads, choked under a flood of multi-gigabyte uploads. And profiling revealed network and EBS I/O bottlenecks that capped throughput well below what the pods could theoretically handle. Those discoveries are now informing the next push: the team says they’re moving even faster in the coming weeks and months. The fact that this entire operation generated only a few dozen GitHub issues and forum threads suggests the bridges are holding. But the real test will be whether the system stays this quiet as they accelerate migrations across the remaining LFS repos — and whether the lessons from bartowski’s migration scale to the rest of the Hub without new surprises.