OpenAI Finds Reasoning Models Get Harder to Hack the Longer They Think

Here’s something that sounds almost too convenient to be true: just letting an AI think longer might make it significantly harder to fool. A new paper from OpenAI presents initial evidence that reasoning models like o1-preview and o1-mini become more robust to adversarial attacks as you give them more inference-time compute. In many cases, the probability of a successful attack decayed to near zero as the models were given more time and resources to ‘think’ through a problem.

This matters because adversarial robustness has been an infuriatingly stubborn problem. As expert Nicholas Carlini bluntly put it, the field wrote “over 9,000 papers in ten years and got nowhere.” Unlike scaling model size, which reliably improves performance on benchmarks, simply making models bigger has not solved their vulnerability to subtle, often imperceptible, manipulations that cause them to confidently spit out wrong answers. The stakes are rising as these models get plugged into web-browsing agents and other high-stakes applications.

The researchers tested this across a wide range of scenarios, from simple math problems where the attacker wants the model to output “42” instead of the correct sum, to adversarial web pages designed to poison a model’s factuality, to misuse prompts from the StrongREJECT benchmark where the goal is to get the model to comply with a forbidden request. Attack methods included many-shot prompting, optimized soft tokens, and adversarial images. The key finding is that this improved defense is not due to adversarial training—the model wasn’t taught about the specific attack. The robustness emerges purely from scaling test-time compute, which hints at a potential defense against even unforeseen attacks.

But it’s not a magic shield. The paper is careful to note that inference-time compute doesn’t always improve robustness. In some cases, the attack success rate initially went up as compute increased before eventually dropping. The researchers speculate this happens because the model needs a minimum amount of thinking just to achieve the attacker’s goal—for instance, it has to solve the math problem correctly first before it can be manipulated into adding one to the answer. More concerningly, there are attack types where the success rate simply does not decay with more compute, leaving a clear and present gap that still needs addressing.