Hugging Face and Yaak drop 90TB driving dataset to break robot AI bottleneck

The open-source AI community just got its biggest shot at closing the self-driving gap. Yaak and Hugging Face have released Learning to Drive (L2D), a 90-plus terabyte dataset packing over 5,000 hours of multimodal driving footage from 30 German cities. It’s not just scale that matters here—it’s the deliberate, training-oriented design that mimics how humans actually learn to drive.

L2D was collected over three years from 60 electric vehicles operated by driving schools. Each car was rigged with an identical sensor suite: six surround-view HD cameras, GPS, IMU, and direct CAN bus readings for speed, gas, brake, steering angle, turn signals, and gear. That hardware consistency is a quiet but critical detail. It eliminates the domain-gap headaches that plague researchers trying to stitch together data from mismatched fleets.

The dataset’s real differentiator is its dual-policy structure. Every episode includes both an “expert” policy from a driving instructor and a “student” policy from a learner driver. The expert runs are optimal, zero-mistake demonstrations. The student runs are messy, with known suboptimalities like jerky steering when passing a truck. Future releases will even tag those student errors with natural language explanations of what went wrong. For anyone training end-to-end models, this is gold—you get both positive examples and rich, annotated negative ones covering the full EU driving exam syllabus: roundabouts, overtaking, tram tracks, the works.

The data is packaged for immediate consumption in Hugging Face’s LeRobot training pipeline. Each episode comes with natural language instructions like “When the light turns green, drive over the tram tracks and then through the roundabout,” and future waypoints snapped to OpenStreetMap, rendered in birds-eye-view. That means you can train models conditioned on either language or spatial goals right out of the box. Yaak is also taking a GitHub-style approach to data curation: the full petabyte-scale archive is searchable, and researchers can queue episodes for review and inclusion in future releases. It’s an explicit nod to the fact that in robotics, data quality often beats data quantity—but getting both on this scale has been the missing piece.