AI Aces Connections But Still Can't Rotate a 3D Object in Its Head

The gap between what AI models memorize and what they actually understand has never been easier to demonstrate than with a handful of puzzles. In late 2024, Columbia University researchers found that even the best models could solve only 18% of New York Times Connections puzzles. By early 2025, some frontier models were cracking them nearly every time. That’s a startling improvement in a matter of months, but it doesn’t mean machines have suddenly learned to think. It means they got better at pattern matching a specific format.

Nowhere is the divide clearer than in spatial reasoning. Mental rotation problems—the kind where you determine whether two images show the same object from different angles—remain a disaster for language models, even those with vision capabilities. Architects and mechanical engineers do this instinctively. LLMs still fail abysmally. For all the industry chatter about world models helping AI understand physical environments, the reality is that today’s systems can’t manipulate a 3D object in their heads the way a child can.

Memory creates a different kind of failure. A 2024 study from Google and the University of Illinois Urbana-Champaign tested models on Knights and Knaves puzzles—logic problems where some characters always tell the truth and others always lie. When researchers introduced slight variations that didn’t match the training data, models whizzed past the differences and defaulted to memorized answers. The same flaw shows up in SimpleBench, a test where questions look like harder problems the models have seen before. Humans spot the trick immediately. Top-tier models trip over themselves.

That’s the uncomfortable takeaway for anyone bullish on AI’s general intelligence. These aren’t obscure edge cases or adversarial attacks. They’re basic cognitive skills that humans develop early. A model that can recite millions of facts but can’t tell you whether two pictures show the same cube is not intelligent in any meaningful sense—it’s a very good database with a plausible-sounding voice attached.