Async RL's dirty secret: 98% of weights don't change between steps, and TRL now exploits that

Every async RL library has a dirty secret: every single step, the trainer ships the entire model to the inference engine. For a 7B parameter model in bf16, that’s 14 GB. For frontier models with a trillion parameters, you’re looking at a terabyte. Per step. Fireworks put a memorable number on this in their recent post: 1024 GiB for a full snapshot, sitting on the critical path while GPUs idle.

It turns out you don’t have to. Between two consecutive RL optimizer steps, roughly 99% of bf16 weights are bit-identical. The actual delta is tiny. Fireworks measured an average delta of just 20.3 GB—about 1.98% of the full model. Cursor’s Composer 2 team told a parallel story, stitching together training and inference clusters in different regions with nothing but a shared S3 bucket and compressed weight diffs.

We just landed a PR in TRL that encodes only the changed elements as a sparse safetensors file, uploads it to a Hugging Face Hub bucket, and tells vLLM to fetch it. On Qwen3-0.6B, the per-step payload drops from 1.2 GB to somewhere between 20 and 35 MB. The trainer publishes “weights ready” and uploads the moment its optimizer step finishes. The inference engine fetches on its own schedule. No shared cluster, no RDMA, no VPN required.

The reason this works so reliably comes down to bf16 arithmetic. A bf16 number has only 7 mantissa bits. At the learning rates RL typically uses—say, 1e-6—the optimizer’s update often falls below the rounding threshold. Adam is whispering, and bf16 literally cannot hear it. The byte representation doesn’t change, so the weight doesn’t change. We ran a full disaggregated training where the trainer sat on one box, vLLM lived in a Hugging Face Space, the Wordle environment lived in another Space, and weights flowed through a single Hub bucket. Two orders of magnitude cheaper, and you can pip install it.