When neural networks teach, interpretability requires patience

Machine learning researchers have long known that neural networks make terrible teachers — not because they fail to educate, but because their methods are utterly inscrutable. A teacher network trained to select the most informative examples for a student model typically converges on strategies that, while effective, look like noise to human eyes. It works. You just can’t understand why. A new paper flips this dynamic on its head with a deceptively simple fix: stop training the teacher and student jointly. Instead, train them iteratively, and the teacher suddenly develops strategies that map cleanly onto human intuition.

The researchers tested this across four conceptual domains — rule-based, probabilistic, boolean, and hierarchical concepts. In each case, the iteratively trained teacher learned to select or generate examples that were not only pedagogically sound for the neural student but also recognizable to people. To prove the point, they ran human experiments showing that these machine-generated teaching examples actually work on humans. The teacher wasn’t just interpretable in theory; it could teach flesh-and-blood learners.

What makes this interesting isn’t just the result but what it says about how joint optimization scrambles explainability. When both networks update simultaneously, the teacher finds shortcuts that exploit quirks of the student’s current state — adversarial perturbations in the form of curriculum design. Break that feedback loop, and something closer to genuine pedagogy emerges. The teacher has to settle into strategies that work robustly, not just against a specific co-evolving partner.

There’s a broader lesson here for AI alignment and interpretability research. We spend enormous effort trying to decode what neural networks have learned after the fact. This work suggests that sometimes the training procedure itself determines whether a system’s behavior will ever be legible to us at all. The question isn’t just what the model knows — it’s whether we built a process that lets understanding crystallize in a human-recognizable form.