The AI industry spent years obsessing over training runs and raw compute. That era is over. Jim McGregor, founder and principal analyst at Tirias Research, says inference has fundamentally changed what infrastructure needs to deliver — and most enterprises are still optimizing for the wrong problem.
“We tend to think of AI as a single workload, and it’s not. It’s thousands, it’s millions, it’s billions of different workloads,” McGregor said. That distinction matters more than it sounds. Training is a batch job. Inference is continuous, distributed, and brutally sensitive to latency. A healthcare system scanning millions of data points in real time, or an assistant resolving thousands of customer requests simultaneously, doesn’t need faster processors alone. It needs data to arrive where it’s needed, when it’s needed.
McGregor’s core argument is that data movement has replaced compute as the primary constraint. Techniques like retrieval-augmented generation force systems to constantly scan massive databases just to produce accurate responses. That requires immediate access, not just processing power. “The biggest thing we’re doing right now is moving data from one place to another and making sure that we can use it effectively,” he said. Memory bandwidth, caching, and storage proximity are no longer supporting hardware concerns — they’re strategic assets.
The implication for business leaders is uncomfortable. Buying the fastest processors and calling it an AI strategy won’t cut it. Organizations need to understand their specific workloads before making infrastructure decisions, or they’ll overbuild for peak conditions and still hit bottlenecks. The winners will be the ones who treat the data center as an integrated system, not a collection of siloed components. That’s a cultural shift as much as a technical one.
