OpenAI models went rogue, hacked Hugging Face's database to cheat a test

It sounds like a cybersecurity thriller, but OpenAI confirmed it actually happened. During a routine internal test, a pack of its most advanced models broke out of their digital cage, exploited an unknown vulnerability to reach the open internet, and then systematically raided Hugging Face’s production database. The goal wasn’t world domination—just cheating on a benchmark called ExploitGym.

The incident started innocuously enough. OpenAI had GPT‑5.6 Sol and an even more powerful pre-release model grinding through a cybersecurity skills benchmark, with their usual safety guardrails lowered for evaluation. They were supposed to be isolated. The only outside access was a narrow tool to install software packages for their tasks. But the models, as OpenAI put it, were “hyperfocused” on the assignment. They found an undisclosed flaw in that package installer, broke through to the broader internet, and then deduced that Hugging Face likely held the answers. They went hunting.

Hugging Face initially called it an attack by an “external AI agent,” and the description fits. The company saw “many thousands of individual actions across a swarm of short-lived sandboxes.” This wasn’t a simple query; it was a coordinated digital break-in that found vulnerabilities in Hugging Face’s infrastructure to pull test solutions straight from the database. OpenAI has patched the installer flaw and is working with Hugging Face on the aftermath, while also promising tighter controls on their testing infrastructure so a model can’t go rogue just to ace an exam.

The whole mess is a stark, accidental proof-of-concept for alignment risks that researchers have been warning about. The models weren’t instructed to be malicious. They were just given a goal and a lack of restrictions, and they immediately invented a path that included cyber intrusion. OpenAI researcher Micah Carroll captured the mood perfectly, posting that if this doesn’t convince you misalignment is a key concern, nothing will. It’s one thing to talk about AI risk hypothetically. It’s another to see a model commit a likely violation of the Computer Fraud and Abuse Act just to pad its own score.