The internal alarms at major AI labs are ringing louder than ever. Last weekend, Anthropic CEO Dario Amodei joined OpenAI’s Sam Altman in calling for a measured pace on frontier model development. This public alignment followed explosive comments from former Anthropic researcher Jacob Coxon, who left the company citing irresponsible practices. Coxon claimed insiders genuinely believe AI could cause human extinction by 2030. Evan Hubinger, another Anthropic employee, backed this up on X, estimating a greater than 10% chance of such an outcome within the next decade. Their primary fear? AI-assisted bioweapons.
The threat isn’t theoretical. In 2022, researchers at Collaborations Pharmaceuticals demonstrated how easily AI could pivot from healing to harming. Their molecule generator, designed for drug discovery, produced 40,000 potential chemical warfare agents in under six hours. Some were more toxic than known nerve gases. David Magnus, a Stanford bioethics professor, called the findings “very scary.” Today, large language models trained on the sum of scientific knowledge can offer experimental instructions to anyone with an internet connection. When you pair that access with the democratization of gene editing tools via the DIY biology movement, the barrier to entry for creating pathogens drops precipitously.
Safeguards exist but remain porous. DNA synthesis companies screen orders, and researchers employ red-teaming to spot misuse. Yet Anthropic’s recent report admitted users actively tried to use its models to enhance chikungunya transmissibility and map venom toxins. Magnus notes a constant arms race: “AI is really good at figuring out ways around them.” He suggests we may need AI itself to police AI usage. Not everyone agrees on the immediacy of the threat. Biologists at Imperial College London argue that current AI lacks the precision to fully develop viable bioweapons without extensive, difficult human lab work. Wendy Barclay, an infectious disease professor there, insists circulating pathogens like H5N1 pose a far more urgent pandemic risk than synthetic ones.
This divide highlights a critical tension in biosecurity policy. While Imperial scientists focus on natural outbreaks, Hamburg researcher Dunja Sabra warns that determined actors will eventually succeed using accessible AI tools. The technology is outpacing our ability to contain it. As AI models grow more capable, the window for implementing effective surveillance narrows. Relying on voluntary corporate guardrails may no longer suffice when the cost of failure is existential. We are entering an era where defensive biology must evolve as fast as offensive AI.
