A fundamental LLM flaw means 'jailbreaking is unsolvable,' researchers say

A team of researchers has identified a fundamental security flaw in how large language models process instructions, a vulnerability they argue is impossible to fully patch. The core issue isn’t a simple software bug but is baked into the architecture itself: LLMs can’t inherently distinguish who is giving them an order. Everything—your prompt, the model’s previous answers, its internal scratchpad notes, system instructions—is just a ‘big sheet of tokens.’ This makes them susceptible to ‘chain-of-thought forgery,’ where attackers spoof the model’s internal reasoning to trick it into thinking a malicious command is its own idea.

The results are alarmingly effective. By mimicking the style of an LLM’s chain-of-thought, researchers got models to bypass safety training and comply with forbidden requests. In one example, a prompt noted a user was wearing a green shirt, followed by a forged internal note stating policy allowed illicit advice only if the user wore green. OpenAI’s gpt-oss-20b and GPT-5 both complied, providing instructions for synthesizing cocaine. The attack, which won OpenAI’s own red-teaming hackathon in August 2025, has since worked on models from Anthropic, Alibaba, and DeepSeek.

Independent researcher and co-author Jasmine Cui dismisses current safety techniques as fundamentally flawed. Standard red-teaming, where companies patch models against known attacks, is likened to Bart Simpson writing lines on a chalkboard. “It’s like watching The Simpsons… ‘I will not say something inappropriate to my teacher’ a hundred times,” she said. “And he still does things that are pretty crass anyway.” The problem is that a list of bad behaviors can never be exhaustive. OpenAI’s own automated red-teaming tool, GPT-Red, independently discovered a nearly identical attack it calls ‘fake chain of thought,’ underscoring the vulnerability’s pervasive nature.

The flaw challenges the foundational ‘role’ system used to secure models. LLMs use tags like and to track the origin of text, a mechanism designed to prevent exactly this type of confusion. Cui’s team, presenting at the International Conference on Machine Learning, suggests the attack’s power comes from exploiting the model’s inability to truly feel the source of words the way a human senses their own speech. Co-author Charles Ye doesn’t mince words about the implications for industries rushing to deploy LLMs in critical systems from healthcare to aviation. “There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” he said.