Hugging Face pushes dataset per-file limits to 500 GB to handle terabyte-scale AI data

If you’re sitting on a massive ML dataset and haven’t shared it because the logistics seemed like a nightmare, Hugging Face just removed a major technical hurdle. The platform is rolling out a backend update, courtesy of the Xet team, that smashes the previous per-file upload cap from 50 GB to a whopping 500 GB. For research teams wrestling with terabyte-scale repositories, that’s the difference between hacking together a complex sharding solution and simply uploading your work.

This isn’t just about raw storage. The Hub’s Dataset Viewer means anyone can poke around your data in a browser before downloading a single byte—full-text search, sorting, and all. I’ve seen the feature in action on the Arxiver dataset, where you can instantly pull up papers by a specific author like Ilya Sutskever across 63,400 rows of Markdown-converted arXiv papers. That kind of zero-friction exploration changes how the community discovers and actually uses shared data.

Compatibility is another quiet superpower here. Hosting a dataset on the Hub makes it immediately queryable with tools data scientists already live in. You can stream a Parquet file directly into a Pandas DataFrame, a Polars query, or a DuckDB SQL console with a single line of code—no custom download scripts, no OAuth tangles. The new SQL Console even runs entirely in-browser and lets you share embeddable query results, which is a genuinely useful feature for collaborative debugging sessions.

Access control is also getting more granular. Beyond simple public or private toggles, the platform supports gated datasets that require users to agree to specific terms before access. For companies or institutions that want to share data with the research community while retaining some semblance of governance, that’s a pragmatic middle ground. The real question is whether researchers who’ve defaulted to dumping data on institutional servers or generic cloud buckets will migrate to a platform where their work gets actual visibility and tooling. Hugging Face is betting the answer is yes, and with Nvidia, Stanford, and NASA already hosting there, the social proof is stacking up.