Transformers hit a wall—these 4 startups are reinventing LLMs from scratch

The transformer—the engine inside every major large language model—is showing its age. First described by Google researchers in 2017, its ‘dense attention’ mechanism sucks up staggering amounts of power and chokes on the very long-context tasks that next-gen models are supposed to master. OpenAI alone is on track to spend $50 billion on compute this year, Greg Brockman has said. That bottleneck has cracked the door open for a new wave of startups betting their futures on architectures that could unseat the incumbent.

The core issue is math. A transformer compares every word in a text with every other word. A 10,000-word document can trigger 50 million multiplications. That quadratic scaling made sense when models were smaller, but as reasoning models scribble chain-of-thought notes to themselves and agents juggle entire codebases, the cost is becoming untenable. ‘The entire AI industry is built on transformers,’ says Justin Dangel, CEO of Miami-based Subquadratic. ‘They are one of the most important innovations in the history of computer science, and they’ve changed the world.’ His company thinks it’s time to move on, and it’s not alone.

Subquadratic is attacking the problem head-on by swapping dense attention for a ‘sparse’ version that calculates only the word relationships that actually matter for a given piece of text. The company claims its model, SubQ, can rival top mainstream LLMs on coding and search benchmarks—a claim that still raises eyebrows among researchers who’ve watched many sparse attention schemes fail to match dense attention’s accuracy. Meanwhile, San Francisco-based Manifest AI is trying an even more radical detour, ditching attention entirely for something it calls ‘power retention,’ which stores only the most task-relevant information.

What makes this moment different is the sheer scale of the prize. The International Energy Agency predicts data-center electricity consumption will double by 2030, making efficiency a hard economic requirement rather than a nice-to-have. If any of these startups—or the others chasing state-space models and liquid networks—can deliver even a 30% reduction in compute cost without sacrificing capability, the incumbents will have no choice but to pay attention. The transformer era isn’t over, but for the first time in nearly a decade, its monopoly is in question.