Hugging Face's 4 PB Parquet problem just got a 100x cheaper fix

Hugging Face hosts nearly 21 petabytes of datasets, and over 4 PB of that is Parquet files. That’s a massive storage bill, and traditional methods choke on Parquet’s internal compression—a single row change can rewrite entire files byte-for-byte. The fix, now live in PyArrow and Pandas, is Parquet Content-Defined Chunking (CDC).

The magic is in how CDC pairs with Hugging Face’s Xet storage layer. Instead of re-uploading a 96.1 MB file when you shuffle rows or add a column, Xet identifies only the changed data chunks. In a demo using a 100,000-row slice of the OpenOrca dataset, re-uploading an exact copy transferred precisely zero new bytes. Even when columns were added or removed, the transfer was often measured in kilobytes, not megabytes.

Developer experience is deliberately simple. You just pass use_content_defined_chunking=True to df.to_parquet() in Pandas or pq.write_table() in PyArrow when targeting an hf:// URI. Under the hood, the Parquet writer restructures data pages to maximize shared chunks, meaning your workflow doesn’t change but your bandwidth bill does.

The implications for iterative data work are immediate. Appending rows, tweaking a column type, or even changing the row-group size—operations that previously forced a full re-upload—now transfer only the delta. For teams constantly iterating on large datasets stored on the Hub, this isn’t a minor optimization. It’s the difference between waiting on a 100 MB transfer and a sub-megabyte one.