TRL just shipped a reinforcement learning trainer that makes PPO look bloated. The RLOO (REINFORCE Leave One-Out) Trainer, built on work from Cohere’s Ahmadian et al. (2024), drops the value model entirely — that’s one less copy of the model sitting in GPU memory. The result: 50-70% less vRAM depending on model size, 2x faster training on 1B models, and up to 3x faster at 6.9B parameters. If you’ve ever watched an OOM error kill a training run at 2 AM, you know exactly why this matters.
Here’s the core difference. PPO loads four models into memory: the policy, a reference policy, a reward model, and a value model. RLOO cuts that to three by eliminating the value model, which PPO needs for Generalized Advantage Estimation. Instead, RLOO treats the entire generated completion as a single action rather than scoring each token individually. A reward signal that PPO would attach only to the EOS token gets distributed across the whole sequence in RLOO — a cleaner formulation that sidesteps the sparse-reward problem that makes PPO so finicky to tune.
It’s also worth remembering that PPO’s implementation details have tripped up even experienced teams. Engstrom et al. (2020) and Huang et al. (2022) both documented how subtle bugs in PPO can silently degrade performance. RLOO’s simpler REINFORCE-based loss removes a chunk of that complexity. The paper’s results show RLOO matching PPO on response win rate judged by GPT-4 while consistently beating offline methods like DPO.
The timing matters. A growing body of evidence — including papers from February and May 2024 — suggests online RL outperforms offline approaches like DPO for alignment. But PPO’s hardware demands kept online RL out of reach for most researchers. RLOO changes that calculus. The question now is whether the open-source community actually picks it up or keeps defaulting to DPO out of habit.
