LinkedIn’s push to build AI agents that handle multi-step recruiting and learning tasks hit a wall when GPT-OSS models simply refused to learn. The team needed a backbone model that could reason over incomplete info and coordinate tools — not just spit out answers. GPT-OSS looked promising on benchmarks but crumbled during actual agentic reinforcement learning.
Training runs on the ReTool task, where models solve math problems with a code compiler, went off the rails fast. Rewards flatlined while KL divergence and entropy exploded. Gradient norms spiked, as shown in their experiments charting the collapse. For context, a Qwen-2.5-32B model trained under identical conditions cruised to significantly higher rewards. Something specific to GPT-OSS was broken.
The culprit? A mismatch in how the Mixture of Experts architecture routed tokens during two forward passes. Proximal Policy Optimization requires the importance sampling ratio between old and current policies to be exactly one during on-policy updates. But the gating network was sending the same state-action pair to different experts on each pass, creating a tiny log-probability discrepancy. That discrepancy falsely triggered PPO’s clipping mechanism, corrupting the entire training signal.
The fix forced both passes to use identical expert routing, restoring the mathematical integrity of the on-policy update. Once the ratio stayed at one as intended, the optimization stabilized. It’s a textbook case of how a subtle architectural detail — stochastic expert assignment — can silently break a reinforcement learning pipeline that works fine on dense models. The solution applies to both the 20B and 120B GPT-OSS variants, though the blog focuses on the smaller model’s results. For teams betting on MoE architectures for agent training, this is a debugging landmark.
