OpenAI’s Habitat platform is quietly doing the heavy lifting behind ChatGPT’s interface, processing over 70 million requests every second. This infrastructure supports more than 1 billion weekly users across nearly 40 geographic regions, managing a staggering 500 petabytes of data. What makes this feat notable isn’t just the scale, but the tool choice: it’s built primarily in Python. While often criticized for performance limitations in high-throughput systems, OpenAI engineers Jon Lee, Chaomin Yu, and Ben Ries detail how they stretched the language to serve as a reliable storage layer.
The system didn’t start as a monolith. In mid-2024, Habitat was merely a client-side Python library designed to simplify database interactions for product teams. It abstracted away the complexities of Azure Cosmos DB, handling schema lookups, encryption, and routing so engineers could focus on features rather than infrastructure. This simplicity drove rapid internal adoption. However, by mid-2025, the library model hit a wall. Coordinating backward-compatible protocol changes across dozens of services became a logistical nightmare, with simple bug fixes or routing updates taking days to roll out safely.
The shift from a library to a centralized service was driven by necessity, not just ambition. The team faced unprecedented year-over-year growth exceeding 10x for three consecutive years. Unlike typical engineering roadmaps that plan for gradual scaling, OpenAI had to make tactical decisions to squeeze efficiency out of their existing stack while fending off capacity crunches. They moved critical logic server-side to reduce the blast radius of regional outages and streamline deployments. This transition allowed them to implement complex sharding and shadowing strategies without forcing every client team to update their code simultaneously.
This evolution highlights a pragmatic approach to technical debt. Instead of rewriting everything in a faster language like Rust or Go immediately, OpenAI optimized their Python implementation to buy time for foundational investments. The result is a distributed system that balances speed and reliability under extreme load. As they prepare to detail their multi-tenancy strategies and Azure Cosmos DB partnership in a future post, the current architecture stands as a testament to refining existing tools before replacing them. It suggests that for many organizations, the bottleneck isn’t the language, but the architectural pattern.
