AI Learns Minecraft by Binging 70,000 Hours of YouTube

Forget painstakingly programmed bots—OpenAI just showed that the best way to train an AI to play Minecraft is to let it binge-watch humans on the internet. The team built a system that learned to perform complex in-game sequences, including crafting diamond tools, by watching 70,000 hours of unlabeled video. The trick wasn’t just observing the footage; it was figuring out which keyboard and mouse movements created those actions. They solved this by first paying contractors for a small dataset that included action labels alongside the video. That data trained an ‘inverse dynamics model’ (IDM) to guess the precise button presses from video alone. The IDM then automatically labeled the massive online dataset, which was used to train a foundation model via behavioral cloning. The results are striking. From scratch, the model learned to chop trees, craft planks, and build a crafting table—a sequence that takes a skilled human about 50 seconds. It even picked up more esoteric skills like ‘pillar jumping’ and hunting for food. When fine-tuned on a small dataset of people building houses, the model progressed further, crafting wooden and stone tools and even rummaging through village chests. The core insight isn’t just about gaming. It’s a proof of concept that the vast ocean of ‘how-to’ videos online can be converted into robot training data, sidestepping the impossible task of manually labeling the precise actions behind every tutorial.