OpenAI's 'Iterated Amplification' trains AI on tasks no human can judge

Most real-world problems don’t come with a neat score counter. You can’t algorithmically evaluate a city-wide transit redesign or the security of a sprawling computer network the way you tally points in Go. Yet machine learning fundamentally needs a training signal — some way to tell the system it’s on the right track. Without one, you get nothing. With the wrong one, you get disaster. OpenAI’s latest safety research, published in a preliminary blog post, tries to bridge this exact gap with a technique called iterated amplification.

The core assumption is refreshingly pragmatic: a human can’t judge the whole mess, but they can usually break a small piece of it into even smaller pieces. Ask someone to “defend this entire corporate network” and they’ll freeze. Ask them to look at one suspicious log line, or to split “defend the network” into “worry about the servers” and “worry about the routers,” and suddenly the impossible becomes a series of manageable steps. The method starts by training AI on these tiny, human-solvable subtasks through direct demonstration. Then you move one level up: humans decompose a slightly larger problem into those same subtasks the AI now handles, and the assembled solution becomes the training data for the next generation of models. Rinse and repeat.

This isn’t just theoretical hand-waving. The team draws a direct line to expert iteration, the engine behind AlphaGo Zero, but highlights a crucial difference. Expert iteration polishes a signal that already exists; iterated amplification builds one from scratch. It also shares DNA with algorithms that decompose problems on the fly during testing, though OpenAI’s approach works in settings where no prior training signal exists at all. The team is upfront about the project’s infancy — they’ve only tested it on simple toy algorithmic domains and haven’t even plugged in a real human as the training signal yet, relying instead on an algorithmic stand-in.

For anyone tracking AI safety, the proposal lands in familiar territory. It’s another attempt, like their work on AI debate, to solve the specification problem: how do you tell a system what you actually want when you can’t even articulate it yourself? The blog post reads less like a product announcement and more like an open notebook from researchers who think they’ve spotted a path worth walking. Whether that path leads anywhere beyond toy problems remains, as they readily admit, an open question.