How OpenAI scaled Kubernetes to 2,500 nodes and survived

Running a Kubernetes cluster at 500 nodes is one thing. Pushing past 2,500 is a parade of cascading failures that the OpenAI infrastructure team has been documenting with the grim enthusiasm of veterans comparing shrapnel wounds. Their journey reveals that the official limits aren’t theoretical—they’re a preview of exactly where your cluster will break.

The first wall came at 500 nodes. Researchers started getting regular timeouts from kubectl, and throwing more Kube masters at the problem only masked the real bottleneck. The culprit was etcd, their central state store. Despite using Azure P30 SSDs rated for 5,000 IOPS, etcd was only squeezing out about 10% of that. The issue wasn’t throughput but latency—network-attached disks introduced 2ms write latency, and since etcd performs sequential I/O, it’s latency-bound rather than throughput-bound. The fix was counterintuitively simple: moving the etcd directory to the local temp SSD on each instance dropped write latency to 200 microseconds. Healthy etcd, happy cluster.

Then 1,000 nodes came. High commit latency returned, and the kube-apiservers were suddenly reading north of 500MB/s from etcd. Prometheus monitoring and audit logging exposed the real offenders—Fluentd and Datadog processes were hammering the apiservers with LIST API calls for Events from every single node. Rate-limiting those polling processes stabilized the load. Another smart move: offloading Kubernetes Events to a dedicated etcd cluster using the —etcd-servers-overrides flag, so event creation spikes couldn’t choke the main etcd instances holding pod and node state.

At this scale, etcd’s default 2GB storage limit became a ticking bomb. Hitting that ceiling caused etcd to reject writes entirely, which cascaded into all Kube nodes failing health checks and the autoscaler deciding to terminate every worker in the cluster. They’ve since increased the quota-backend-bytes flag and added a sanity check preventing the autoscaler from ever terminating more than 50% of the cluster at once. The team also flipped the default kube-scheduler policy on its head—instead of spreading pods evenly, they pack them tightly so idle nodes can be terminated and large pods schedule faster. That optimization broke KubeDNS when certain pods ended up hosting 10+ copies, but the team’s willingness to share these war stories is what makes this post invaluable for anyone who thinks they’ve outgrown Kubernetes.