Nvidia's Huang sees 'AI factories' but cloud storage is already buckling

Jensen Huang is out there talking about “AI factories” pumping out intelligence, and he’s not wrong about the direction. But the conversation is stuck on GPUs and model size while ignoring a disaster unfolding one layer down. The shift from chatbots to autonomous AI agents is turning inference from a compute problem into a massive, unpredictable data problem, and the storage layers inside AWS, Azure, and Google Cloud were never built for this.

Think about how we’ve designed data systems for two decades. They’re tuned for humans who click, read, and sip coffee. Traffic is predictable, cacheable, and easy to average out. Now replace that human with an autonomous agent running a ReAct loop. It fires a query, ingests context, realizes it’s missing something, and fires three more queries in parallel — all in milliseconds. Multiply by thousands of concurrent agents, and you’ve got what engineers are calling “OLTP++”: massive read spikes with access patterns that look like noise. Peak load is the only load that matters. If your capacity planning relies on CloudWatch averages and historical CPU, you’re already in trouble.

Then there’s the vector database mess. Everyone is obsessing over prompts and fine-tuning, but when a RAG application moves from a Jupyter notebook to production, the bottleneck isn’t the LLM or your Python code. It’s the data path — index scans, embedding fetches, scatter-gather latency. Throwing AWS RDS read replicas at the problem doesn’t fix it. It just relocates the bottleneck. Engineering teams that treat replicas as a scaling strategy are, as one expert put it, “exactly one traffic peak away from a catastrophic post-mortem.”

And AWS EBS? It’s a workhorse, but it’s bound by strict IOPS caps and burst buckets that don’t care about your SLAs. A surge of agent-driven traffic will drain those burst credits in minutes. Latency spikes from 1 millisecond to 50. Worker threads exhaust. The stack locks up. You can’t just provision your way out of it because you’ll hit the physical limits of what a single EC2 instance can push. The architecture assumption that storage will just keep up is quietly breaking, and the people who feel it first will be the on-call engineers staring at graphs that went vertical before going dark.