Reinforcement Learning from Human Feedback (RLHF)

Reinforcement Learning from Human Feedback (RLHF) is a key technique for aligning large language models with human preferences. It was notably used by OpenAI to train ChatGPT and InstructGPT.

The RLHF process involves three steps:

1. Supervised Fine-Tuning: The model is fine-tuned on human-written demonstrations
2. Reward Model Training: Humans rank model outputs, and a reward model learns to predict these preferences
3. Reinforcement Learning: The language model is optimized to maximize the reward model’s score

RLHF has been instrumental in making LLMs more conversational, helpful, and less likely to produce harmful content. Variations like Constitutional AI and Direct Preference Optimization (DPO) have since emerged as alternatives.