PipelineRL ditches value functions, matches Open-Reasoner-Zero with inflight weight updates

The team behind PipelineRL just dropped a compelling rebuttal to one of reinforcement learning’s most stubborn trade-offs. In conventional RL training for language models, you either get high throughput or on-policy data — but not both. Batching inference efficiently means generating rollouts that are already stale by the time the optimizer uses them. PipelineRL’s fix is surprisingly straightforward: update the model weights on the inference servers without ever stopping them.

They call these inflight weight updates. After each optimizer step, the system pauses inference just long enough to broadcast new weights via NCCL, then resumes. The KV cache contains keys and values computed with the previous model version, which sounds like a recipe for instability. It isn’t. Training runs stayed stable, and the team reports that trust region clamping, overlong sequence filtering, and reward shaping were all unnecessary. They normalized loss simply by the number of sequences in the batch, used no KL penalty, and skipped entropy bonuses entirely.

The results back up the simplicity. A 7B model and a 32B model trained on the Open-Reasoner-Zero dataset matched or exceeded that system’s scores on AIME 2024 and MATH 500. This matters because Open-Reasoner-Zero relies on a value function — a heavier, more complex approach — while PipelineRL runs a stripped-down variant of GRPO. As one researcher put it, inflight weight updates let the inference server “constantly maintain the optimal batch size while simultaneously ensuring data remains on-policy or near on-policy.”

The architecture is deliberately modular, built around clear API contracts between inference and training components. Inference software — whether SGLang, vLLM, or Nvidia Dynamo — needs to expose just three endpoints: one for process group initialization, one for triggering weight updates, and a standard chat completion interface. That separation means teams can swap in faster inference engines or trainers as they emerge, without rewriting the orchestration layer. If the stability holds at larger scales, this could lower the barrier for groups that want competitive reasoning performance without building the full RL infrastructure that systems like Open-Reasoner-Zero demand.