OpenAI Scales Kubernetes to 7,500 Nodes—Without Breaking

OpenAI just published a detailed breakdown of how it scaled its Kubernetes clusters to a staggering 7,500 nodes, an infrastructure feat that directly supports the training of giant models like GPT-3 and DALL-E. That’s a jump of 5,000 nodes since their last public scaling post, and it required the team to fundamentally rethink core assumptions about networking, scheduling, and the very nature of their workloads.

Here’s the first thing to understand: an AI research lab’s Kubernetes isn’t your Kubernetes. OpenAI’s workloads are dominated by massive MPI jobs where a single pod often takes over an entire physical node to leverage NVLink and GPUDirect. Scheduler strain is therefore spiky and low, not constant. Bin-packing, the bane of many a cloud-native operator, isn’t their problem. Instead, their headaches were at the network layer. The team found that Flannel simply couldn’t keep up as the cluster swelled, so they ripped it out. They moved to native pod networking using Azure VMSS IP configurations and relevant CNI plugins, sidestepping the route-table limitations that crippled their previous approach.

Ditching the overlay network for alias-based IP addressing wasn’t just a performance play. With roughly 200,000 IP addresses potentially floating around, avoiding encapsulation kept the entire setup brutally simple. No packet fragmentation worries, no ambiguity for traffic monitoring—just host-level throughput. They’re now using iptables tagging on the host to track network resource usage per namespace and pod, a transparency boon for researchers who can finally visualize their network patterns.

But the post isn’t just a victory lap. It’s a frank admission of the remaining thorny problems that come with operating at this scale, where the ephemeral, research-driven nature of the workloads means a new usage pattern can shatter your careful assumptions at any moment. The team considers pods semi-stateful; a single pod dying can kill an entire multi-hundred-pod MPI job, forcing a disruptive restart from a checkpoint. That fragility drives a relentless focus on infrastructure quality for an application base that is, by definition, unstable and experimental. It’s a tightrope walk that few others have to perform.