AI tutors still can't master the one skill that defines great teaching: knowing when to shut up

Most benchmarks for AI tutors reward a single, simplistic behavior—like never giving away an answer. Real teaching is messier. It’s a judgment call made hundreds of times per session: does this specific kid, stuck on this specific problem, need a scaffolded hint or a push to reason harder? A new evaluation framework called TutorMoments suggests that even the most advanced language models are pretty bad at making that call.

The research, built from over 1,500 teacher-annotated moments in real K-12 math tutoring transcripts, tested how models act when dropped into a session at a critical decision point. The default instruction to “tutor well” produced a predictable result: the LLMs over-helped. They rushed to explain concepts and lay out steps, robbing a simulated student of the productive struggle that learning science consistently ties to deeper understanding. This isn’t a minor glitch—it’s a direct collision between how models are trained (to be maximally helpful assistants) and what effective tutoring demands (calibrated, strategic withholding).

Researchers found that spelling out the scaffolding-versus-rigor trade-off directly in the model’s prompt improved performance. But it didn’t close the gap with human tutors, who varied their approach moment-to-moment in ways the models couldn’t reliably replicate. The LLMs still differed wildly in their consistency. One moment an AI might nail a Socratic question; the next, it would spoon-feed the solution like a search engine, completely misreading the student’s zone of proximal development.

This framework matters because it shifts the evaluation from rigid checklists to context-dependent decision-making. The team is releasing the de-identified transcripts, annotations from 27 experienced teachers, and the replay pipeline. For anyone building an AI tutor, it’s a sharper stress test than standard accuracy benchmarks. The uncomfortable takeaway? Making a model that can explain calculus is the easy part. Making one that knows when to be quiet is a fundamentally different challenge—and one that isn’t solved by simply scaling up parameters.