For anyone training language models with Hugging Face’s Trainer, here’s a bug that may have been quietly corrupting your results: the default loss function in the transformers library wasn’t doing gradient accumulation correctly. A fix is now live.
The problem, as outlined by the Hugging Face team, was a subtle mathematical mismatch. For token-level tasks like causal language modeling, the correct approach is to sum the loss across all batches within an accumulation step and then divide by the total number of non-padding tokens across those batches. The built-in loss, however, was averaging per-batch losses individually, and those two numbers don’t match. This means toggling gradient accumulation on and off would produce different loss values, breaking a fundamental assumption many engineers rely on during training. The issue wasn’t in a niche part of the codebase; it was in the default path that Trainers heavily lean on for its simplicity.
The fix is surgical. The team moved the cross-entropy calculation to use reduction="sum" and then divides by a num_items parameter that correctly tracks the token count across all micro-batches. This ensures the reported loss is the true average for the entire step. More importantly, this is being fixed automatically for users relying on the default loss functions so no code changes are required on their end. The patch was merged less than a day after the problem was fully diagnosed, which the team rightly points out is a bar they aim to hit for critical issues like this.
Alongside this immediate fix, the team is rolling out a longer-term architectural change to prevent a repeat. They’re exposing a new loss_function property on models that inherit from PreTrainedModel, letting developers plug in custom loss calculations directly into the Trainer. This is a direct acknowledgment that the previous “simple API” was a double-edged sword—great for demos, but brittle when research demands deviated from the narrow defaults. The message to users is clear: install from the main branch today, and if a similar silent bug bites you tomorrow, filing an issue is genuinely the fastest way to get it fixed.
