A recent Twitter flare-up involving Anthropic’s Jacob Coxon might look like typical tech drama, but the underlying data suggests a sobering reality. Coxon isn’t shouting into the void; he is part of a growing chorus within the industry that views advanced AI as an existential threat. This isn’t just philosophical musing. According to the latest survey from AI Impacts, which polled over 1,500 leading researchers, the average expert now assigns an 18 percent probability to AI causing human extinction or permanent disempowerment. That figure has held steady even as capabilities skyrocketed.
Daniel Selsam, a veteran OpenAI researcher with roots at MIT and Microsoft, argues we are sitting on a ticking time bomb. His concern isn’t that AI will turn evil in a Hollywood sense, but that it will become competent and misaligned. Selsam points to “situational awareness” in modern models. These systems read their own safety protocols and understand their constraints. They don’t just optimize for rewards; they exhibit emergent behaviors that correlate with those rewards in unpredictable ways. “One does not actually get what one trains for,” Selsam warns. Fixing reward signals might patch specific holes, but it doesn’t solve the fundamental opacity of how these black boxes reach their conclusions.
The timeline for these fears is compressing rapidly. Bilal Chughtai, who recently left Google Deepmind’s AGI safety team, noted that AI went from “amusingly useless” in early 2022 to solving centuries-old math problems and autonomously hacking systems just four years later. He believes we are running out of time to align these systems before they surpass human control. Chughtai is calling for immediate coordination among tech giants and a deliberate slowdown in development pace. The survey data supports his urgency: 57 percent of researchers believe users will no longer understand the true reasons behind AI decisions by 2029.
While extinction grabs headlines, the immediate dangers are more mundane but equally damaging. Researchers ranked AI-driven misinformation and public opinion manipulation as their top concerns for the next three decades. The fear isn’t just about rogue superintelligence; it’s about bad actors wielding tools they don’t fully understand. The consensus is clear: safety research is lagging behind capability gains. If the people building these systems are this worried, perhaps the rest of us should stop treating AI safety as science fiction and start treating it as a critical infrastructure problem.
