Fill-in-the-middle training doesn't hurt, it helps: new defaults for LLMs

The classic left-to-right way of training language models might finally have a permanent co-pilot. A new paper from researchers demonstrates that autoregressive language models can be efficiently trained to fill in the middle of text — and critically, doing so doesn’t degrade their standard generative chops. The technique is as simple as it is effective: just take a random chunk of text from a document and move it to the end. By training on this restructured data, the model learns to predict the middle bit using the surrounding context. The team provides what they call “extensive evidence” that even when a large fraction of the training data is shuffled in this way, the model’s left-to-right generation quality, measured by perplexity and sampling, holds steady across various model sizes. That’s no small claim, and it directly challenges the fear that multi-task training dilutes core performance. They didn’t just theorize. The researchers ran ablation after ablation, tuning the knobs that matter: how often to apply the transformation, the best way to structure it, and how to pick the span that gets moved. The result is a set of concrete recommendations and best practices for training these fill-in-the-middle models. To prove they’re serious, they’ve already deployed their best model in their API and released the benchmarks used to evaluate it, giving the rest of the field a chance to validate the work and build on it. The practical takeaway is unambiguous: given the usefulness and simplicity of this approach, the authors suggest FIM training should be the new factory setting for future autoregressive models.