GPT-4 can now explain what every neuron in GPT-2 is doing

Here’s a problem that has kept AI researchers up at night: you build a massive language model, it works brilliantly, but you have almost no idea what’s happening inside it. Individual neurons fire in response to text, but figuring out what each one actually represents traditionally required a PhD staring at spreadsheets of activations. That doesn’t scale when your model has hundreds of billions of parameters.

OpenAI’s latest interpretability experiment asks a smarter model to do the staring. They used GPT-4 to automatically generate natural language explanations for the behavior of every one of GPT-2’s 307,200 neurons, then had GPT-4 score those explanations against the neuron’s actual activation patterns. The team is now open-sourcing the entire dataset, the visualization tools, and the code that uses publicly available models on the API. The ambition here is meta: this is alignment research that automates alignment research itself, an approach that gets more viable as models get smarter.

The results are honestly mixed — and OpenAI is upfront about that. Over 1,000 neurons received explanation scores of at least 0.8 out of 1, meaning GPT-4 rated its own work as capturing most of a neuron’s behavior. But the vast majority of explanations scored poorly, especially in larger models and deeper layers where representations get tangled. Human-written explanations still beat GPT-4’s best attempts, a gap the paper acknowledges without flinching.

What makes this more than a curiosity is the iterative potential. The team found they could boost scores by having GPT-4 generate counterexamples to stress-test its own explanations, then revise. Bigger explainer models produced better scores, and tweaking the activation functions of the model being explained also helped. The current method only describes what a neuron responds to in the input text — not what it does downstream or the mechanism producing that behavior — so it’s more correlation than causation. But for a field that has treated model internals as impenetrable, having even rough labels on every neuron marks a genuine shift in what’s possible.