Hugging Face just published a rare empirical head-to-head of three reinforcement learning-free alignment methods—DPO, IPO, and KTO—and the results carry real weight for anyone fine-tuning chat models. The team trained OpenHermes-2.5-Mistral-7B and Zephyr-7b-beta-sft on two preference datasets, Intel’s orca_dpo_pairs and ultrafeedback-binarized, then measured performance with MT-Bench. That’s a solid setup: two models, two datasets, and a benchmark that actually correlates with human judgment.
There’s a correction worth flagging before anyone cites the original findings. The initial implementation of IPO in TRL was wrong—the loss over completion log-likelihoods was summed instead of averaged. After consulting with the IPO paper’s authors, Hugging Face pushed a fix and re-ran everything. The revised results now align with the original research: IPO performs on par with DPO and beats KTO in the paired preference setting. That’s a meaningful shift from whatever the first version showed, and it’s a reminder that implementation details in these libraries can quietly distort conclusions.
The deeper question the post tackles is which method a practitioner should actually reach for. DPO remains the default choice and for good reason—it’s simple, well-tested, and powers models like Zephyr and Intel’s NeuralChat. But DPO overfits quickly, which is exactly why Google DeepMind developed IPO with its regularization term. KTO, from ContextualAI, takes a different path entirely: it works with individual 👍/👎 labels rather than paired preferences, which are dramatically cheaper to collect in production chat UIs.
The Inblix angle here: the fact that IPO needed a bug fix and a full re-run before matching DPO tells you how young this tooling still is. Teams building on TRL should treat alignment library implementations as suspect until they’ve validated against the original papers. The open-source configs are available in the alignment-handbook repo if you want to reproduce the sweep yourself—and given how much the results changed after one fix, you probably should.
