Transformers hit a wall, so 4 startups are chasing the next great LLM architecture

Nine years after Google researchers gave the world the transformer, the architecture that powers every major large language model is becoming a bottleneck. Will Douglas Heaven reports that the dense attention mechanism at the heart of transformers gets cripplingly expensive as text length grows, and they struggle to keep track of large amounts of information at once. The bigger the model, the more painful the problem.

Four startups are now betting on entirely new approaches that could make LLMs faster, more efficient, and possibly smarter. The details are under wraps in a broader look at what comes next, but the direction is clear: the industry is starting to treat the transformer not as the final word, but as a foundation that needs replacing.

This matters because transformer inefficiency is quietly shaping who gets to build frontier models. If attention mechanisms remain quadratic in cost, only the best-capitalized labs can afford to push scale. A breakthrough architecture wouldn’t just be an academic curiosity — it would redraw the competitive landscape overnight.

At a Schmidt Sciences AI2050 convening in Mountain View, meanwhile, university researchers described a strange and anxious moment. Grace Huckins writes that academic AI scientists, who make up most of the fellowship, are negotiating a reality where compute budgets and talent pipelines are increasingly controlled by industry. The questions they’re asking are uncomfortable: Can universities still set the research agenda when they can’t afford to run the experiments? The transformer’s successor might come from one of those startups — or it might come from a lab that the current economics are trying to squeeze out.