OpenAI teaches o-series models to read the rulebook before answering

OpenAI just flipped the script on AI safety training. Instead of having models guess the rules from thousands of labeled examples, they’re now literally handing them the rulebook. The new method, called deliberative alignment, directly ingrains the text of OpenAI’s internal safety policies into the model’s reasoning process — and it appears to work dramatically better than anything we’ve seen before.

The core problem with existing approaches like RLHF has always been that they’re a guessing game. A model sees a bunch of examples showing what humans consider safe or unsafe and has to reverse-engineer the underlying principles. That’s inefficient and leaves blurry decision boundaries, which is why models still occasionally comply with harmful requests or over-refuse perfectly innocent ones. Deliberative alignment bypasses this entirely by teaching the model the actual safety specifications in natural language and training it to deliberate over them through chain-of-thought reasoning before responding.

The results on o1 are striking. OpenAI reports the model dramatically outperforms GPT‑4o across a range of safety benchmarks, saturating performance on many datasets that previously tripped up even the most robust models. What’s particularly clever is the training pipeline: they first train a helpful-only model with zero safety data, then build a dataset where completions reference the specifications by temporarily injecting them into system prompts, and finally use reinforcement learning with a reward model that also has access to the policies. No human-labeled chain-of-thought data required.

This feels genuinely new. Methods like Constitutional AI use specifications only to generate labels — the model never sees the constitution itself. Self-REFINE adds reasoning at inference time but constrains it to predefined paths. Deliberative alignment is the first to directly teach the policies and let the model retrieve and apply them dynamically. It’s a promising demonstration that advancing raw reasoning capabilities doesn’t have to come at the expense of safety. The two can actually reinforce each other.