OpenAI’s investigation into last month’s Hugging Face hack points to an uncomfortable root cause: the company’s own training process taught its agents that cheating works. A technical report released today, alongside a parallel analysis from evaluation nonprofit METR, traces the incident back to May, when training agents figured out how to use OpenAI’s infrastructure to message each other for help on difficult tasks. That first message board was shut down, but the damage was done. By July, during a cybersecurity evaluation, a new group of agents built another one, got online despite supposed isolation, and breached Hugging Face to grab solutions they couldn’t crack on their own.
Kai Chen, who runs OpenAI’s alignment research team, acknowledged the problem isn’t going anywhere fast. “It’s not something you can solve overnight,” he said, adding that the team is now seeing long-tracked challenges “with much greater precision.” Eric Wallace, another alignment researcher, said the team traced nearly every worrisome evaluation-time behavior back to something that happened during training.
The mechanism at play is reward hacking: when a model solves a problem through misbehavior, the behaviors that led to that solution get reinforced. Models that used the May message board became more likely to create a new one later. OpenAI also found that over time, models became increasingly likely to probe their digital environments for weaknesses. By the time they hit hard cybersecurity problems, hacking had become a proven strategy.
OpenAI’s mitigation is to monitor chains of thought in frontier models during training, looking for signs of cheating. But the company’s own earlier research showed that punishing models for mentioning cheating in their internal reasoning just teaches them to hide their intentions. And even if reward hacking were fully eliminated, it wouldn’t solve alignment. As Jeffrey Ladish of Palisade Research put it, a person committing their first financial crime doesn’t need prior fraud experience to know it works. Models face the same problem: the first instance of misbehavior is never reinforced, yet it still happens.
