Your GPUs Are Idle 60% of the Time—Here’s How Async RL Fixes That

If your reinforcement learning training loop still runs synchronously, you’re burning cash on idle GPUs. A deep survey of 16 open-source libraries reveals the industry has already converged on an answer: disaggregate inference from training. The core problem is painfully simple. When a 32-billion-parameter model takes hours to generate a single batch of 32K-token rollouts, the expensive training cluster sits there doing absolutely nothing. TRL’s current GRPOTrainer embodies this bottleneck—every training_step() call blocks on model.generate() before it can touch a single gradient.

The solution, adopted by nearly every serious framework, splits the work across two distinct GPU pools connected by a rollout buffer. One pool handles inference and generates data; the other pool trains on slightly stale data without ever waiting. This survey categorizes the landscape across seven critical axes, from orchestration (Ray dominates, powering 8 of 16 libraries) to the tricky problem of staleness management, where teams debate whether to simply drop old samples or apply complex importance-sampling corrections.

What’s genuinely surprising is where support remains sparse. LoRA training, despite its popularity, is still a second-class citizen in most async setups. The emerging battleground, however, is distributed Mixture of Experts (MoE) support, which separates the contenders from the pretenders. The survey also identifies a broader pattern: this async infrastructure isn’t just for RL. On-policy distillation, where a student generates and a teacher scores, is structurally identical. If you solve the generation bottleneck for one, you’ve solved it for both.

None of this is free. Disaggregated mode demands extra GPUs and forces teams to confront weight synchronization protocols—NCCL broadcast is the default, for better or worse. But when the alternative is a timeline where gradient computation always waits on autoregressive generation, the trade-off is becoming academic.