AI Pulse by Inblix

Hugging Face now lets you query 12.6M-row datasets with SQL in your browser

Hugging Face Blog · Sep 17, 2024 · 2 min read · Read original article →

Curated by the Inblix editorial team


Featured image for article: Hugging Face now lets you query 12.6M-row datasets with SQL in your browser

Hugging Face just flipped a switch that makes exploring massive datasets feel almost frictionless. They’ve rolled out a SQL Console on every dataset hosted on the Hub, and it runs entirely in your browser. No servers, no Python environment, no waiting for a notebook kernel to spin up.

Under the hood, this is powered by DuckDB’s Web Assembly (WASM) build. Most datasets on the platform are already stored as Parquet files — a columnar format that’s both efficient and fast — and the console automatically creates queryable views from those files. If a dataset isn’t in Parquet, the first 5GB gets auto-converted. The result is a full SQL environment that supports the complete DuckDB syntax, including functions for regex, JSON, lists, and embeddings. As the team puts it, “the SQL Console creates views for you to query based on your dataset splits and configs.”

I tested this against their demo query on the OpenCo7/UpVoteWeb dataset, which clocks in at 12.6 million rows. A simple filter query returned results in under 3 seconds. That’s genuinely impressive for an in-process database engine constrained by a roughly 3GB memory limit in the browser. It won’t replace your entire data pipeline, of course. Complex joins on billion-row tables will hit that memory ceiling hard, and DuckDB WASM doesn’t yet support the hf:// protocol for querying datasets directly from the lake — a feature gap that power users will notice immediately.

But the practical upshot is clear: you can now prototype data transformations in seconds that used to require a separate script. The team demonstrates this by converting an Alpaca-format dataset into a conversational format using a SQL query with struct_pack and CASE statements — a task developers typically handle with a Python pre-processing step. The query runs in-browser, and the results export directly to Parquet. Each query is also shareable via a link, which makes debugging and collaboration far less painful. It’s a smart move that lowers the barrier to entry for anyone who wants to understand what’s actually inside a dataset before committing to training or fine-tuning a model. Just don’t expect it to replace your Spark cluster.

💡 Key Takeaways

  1. Every Hugging Face dataset now includes a browser-based SQL console powered by DuckDB WASM, requiring zero dependencies or backend setup.
  2. The console handles datasets up to roughly 3GB of memory and can query 12.6 million rows in under 3 seconds for simple filters.
  3. Users can export query results to Parquet and share public dataset queries via a link, streamlining collaboration and prototyping.
  4. DuckDB WASM lacks hf:// protocol support, so it cannot query datasets directly from remote storage — a notable current limitation.

Keep reading: See related articles below for more coverage on this topic.

Get smarter about AI

The sharpest AI news, curated daily. Delivered free to your inbox.

Learn more

Glossary terms

← Back to all articles