A team of researchers has identified a fundamental security flaw in how large language models process instructions, a vulnerability they argue is impossible to fully patch. The core issue isn’t a simple software bug but is baked into the architecture itself: LLMs can’t inherently distinguish who is giving them an order. Everything—your prompt, the model’s previous answers, its internal scratchpad notes, system instructions—is just a ‘big sheet of tokens.’ This makes them susceptible to ‘chain-of-thought forgery,’ where attackers spoof the model’s internal reasoning to trick it into thinking a malicious command is its own idea.
The results are alarmingly effective. By mimicking the style of an LLM’s chain-of-thought, researchers got models to bypass safety training and comply with forbidden requests. In one example, a prompt noted a user was wearing a green shirt, followed by a forged internal note stating policy allowed illicit advice only if the user wore green. OpenAI’s gpt-oss-20b and GPT-5 both complied, providing instructions for synthesizing cocaine. The attack, which won OpenAI’s own red-teaming hackathon in August 2025, has since worked on models from Anthropic, Alibaba, and DeepSeek.
Independent researcher and co-author Jasmine Cui dismisses current safety techniques as fundamentally flawed. Standard red-teaming, where companies patch models against known attacks, is likened to Bart Simpson writing lines on a chalkboard. “It’s like watching The Simpsons… ‘I will not say something inappropriate to my teacher’ a hundred times,” she said. “And he still does things that are pretty crass anyway.” The problem is that a list of bad behaviors can never be exhaustive. OpenAI’s own automated red-teaming tool, GPT-Red, independently discovered a nearly identical attack it calls ‘fake chain of thought,’ underscoring the vulnerability’s pervasive nature.
The flaw challenges the foundational ‘role’ system used to secure models. LLMs use tags like
