Hugging Face launches S3-style Buckets to fix why Git stinks for ML storage

Let’s be honest: using Git to manage machine learning artifacts has always been a square-peg-round-hole situation. Checkpoints, optimizer states, raw datasets—these are massive, mutable, and constantly overwritten. Git chokes on them. Hugging Face is finally acknowledging this with Storage Buckets, a new non-versioned object storage container that acts a lot like S3 but lives directly on the Hub under your namespace.

What makes this genuinely interesting isn’t just the Git alternative—it’s the backend. Buckets run on Xet, Hugging Face’s chunk-based deduplication engine. That means if you upload a processed dataset that’s mostly similar to the raw version, or successive model checkpoints where most weights are frozen, Xet recognizes the duplicate chunks and skips them entirely. You save on bandwidth, transfers finish faster, and for Enterprise customers, billing is calculated against the deduplicated footprint. It’s a smart architectural fit: training pipelines naturally produce families of overlapping artifacts, and Xet was built to exploit exactly that redundancy.

Speed-conscious teams will zero in on the pre-warming feature. By default, Buckets live wherever the Hub does, which isn’t ideal when your compute is in a specific AWS or GCP region. Pre-warming lets you declare where your hot data should reside before jobs start, cutting out the cross-region latency that kills distributed training throughput. Hugging Face is starting this partnership with AWS and GCP, with more providers promised.

You can spin up a bucket in under two minutes with the hf CLI—create it, sync a directory, dry-run the plan first if you’re cautious—or use the Python API in huggingface_hub v1.5.0. The loop is straightforward: keep your messy, iterative working data in a Bucket, and save the clean, versioned repo for when you’re actually ready to publish. It’s a pragmatic separation that should have existed years ago.