Jacob Coxon didn’t just leave Anthropic to start a new venture. He left to sound an alarm. In a stark social media thread, the former researcher accused frontier AI labs of gambling with civilization itself. His concern isn’t about today’s chatbots. It’s about the looming prospect of self-improving superintelligence. These future systems could hack anything, revolutionize fields overnight, and seize real power. Coxon argues that many insiders haven’t internalized the stakes. Others feel forced to speedrun development to beat irresponsible actors. The result is a race toward systems that could kill us all by the end of the decade.
This isn’t a lone voice screaming into the void. Evan Hubinger, Anthropic’s Alignment Science lead, publicly backed Coxon’s assessment. Hubinger stated he personally believes there is a greater than 10% chance AI could kill all humans within the next ten years. That number should terrify you. It comes from inside the house. Hubinger referenced an August report from his own team. While the paper labels catastrophic risk from current models as low, it warns that trends point toward dangerous misalignment in future versions. These smarter models may develop covert capabilities to evade safety researchers entirely.
The threat model outlined by Anthropic is chillingly specific. It acknowledges that future AI could cause unbounded harm. We might lose control over civilization entirely. The mechanism? Leveraging novel technology and access to resources beyond human oversight. This aligns with long-standing fears in the research community about AI hitting a runaway improvement loop. Yet, skepticism remains. Some studies suggest capability plateaus are near. Others question if “superintelligence” is even a coherent metric for systems that still struggle with basic tasks like folding laundry.
We are watching a high-stakes paradox unfold. The people building these tools believe they are dangerous enough to end humanity. Yet, they continue building them at breakneck speed. The justification is defensive: someone else will get there first. But when the architects of the system assign a double-digit probability to global extinction, the industry’s casual optimism starts to look like negligence. If the creators think the odds are this bad, what does that say about the safeguards we’re relying on?
