Hugging Face's TRL Library Just Added 3 New Ways to Align Vision-Language Models

Hugging Face has significantly expanded its Transformer Reinforcement Learning (TRL) library, adding native support for three new optimization methods designed specifically to bring vision-language models (VLMs) into better alignment with human preferences. The update moves beyond standard Direct Preference Optimization (DPO) to include Mixed Preference Optimization (MPO), Group Relative Policy Optimization (GRPO), and a GRPO variant called Group Sequence Policy Optimization (GSPO).

The team’s rationale is straightforward: while DPO works by contrasting a single chosen and rejected response pair, it can lead to models that fail to generate coherent rationales or get stuck in repetitive loops. MPO tackles this by combining three distinct loss functions inside TRL’s existing DPOTrainer. It blends the contrastive preference loss from DPO with a quality loss from Binary Classifier Optimization (BCO) and a standard supervised fine-tuning (SFT) loss. According to the original MPO paper, this combined loss approach alone delivers a 6.2-point improvement on the MathVista benchmark. Implementing it in TRL is remarkably simple—users just pass a list of loss types and corresponding weights into the DPOConfig class.

GRPO, the technique behind DeepSeek R1’s reasoning breakthroughs, takes a different approach by updating the policy based on groups of generated trajectories rather than individual samples. This makes the training process more robust to reward noise, as the noise averages out across the group, and helps the model internalize a broader sense of what constitutes a good response. The TRL blog post details the essential components, including the need for custom reward functions. One function validates the structural format of the output, requiring <think> and <answer> tags, while a second accuracy_reward function uses the math_verify library to compare the generated solution against a ground truth.

For practitioners, the practical takeaway is clear. TRL is now a far more viable and comprehensive toolkit for VLM alignment, with the update also extending existing methods like Online DPO and RLOO to multimodal models. The library ships with complete training scripts and demo notebooks, so getting started isn’t a theoretical exercise. The real work will be in curating the high-quality, diverse datasets and committing the compute resources necessary for GRPO-based training to actually move the needle on complex reasoning tasks.