Supervised fine-tuning has a dirty secret when it comes to structured OCR: it can’t reliably stop models from spiraling into gibberish repetition loops. A new paper from the team behind DharmaOCR shows why, and the fix is surprisingly straightforward. They applied Direct Preference Optimization — a technique nearly always used for chatbot alignment — and used the model’s own degenerate outputs as the ‘rejected’ examples. No human feedback required. The task was objective, but the preference signal was crystal clear: a correct transcription is chosen; a repetition loop is rejected.
Across every model family tested, a second DPO stage after SFT reduced text degeneration rates. No exceptions. The average reduction was 59.4%, with one model seeing an 87.6% drop. This isn’t about tweaking a temperature dial at inference time. The paper argues that those are band-aids, only containing the symptom without touching the underlying probability distribution that creates the so-called ‘attractor’ state. A model gets trapped in a loop because its training objective has concentrated probability mass in a way that makes the same next token seem overwhelmingly likely, step after step.
SFT trains token by token, never penalizing a completion-level disaster like a repetition loop. DPO inverts that logic, evaluating the full, finished output as chosen or rejected. The curious outlier in their data proves the rule: one model family saw its degeneration rate spike from 0.60% to 3.23% after SFT, before DPO dragged it back down to 1.41%. It’s a single data point, but it underscores that task-focused fine-tuning alone can actually make the problem worse by pushing the model into new, failure-prone regions of the distribution.
The implication is a fundamental rethinking of training pipelines for structured generation. If your model’s failure mode is consistent and detectable, it’s not just an error to log — it’s a free, high-signal training set. The question isn’t whether DPO can align a model to human values, but what other stubborn, non-subjective failure modes it can directly attack.
