Reinforcement Learning from Human Feedback has quietly become the most important acronym in AI, and most people using ChatGPT have no idea what’s happening under the hood. The core problem: how do you train a model to produce text that’s genuinely good when ‘good’ depends entirely on context? A story needs creativity. A medical answer needs accuracy. Python code needs to actually run.
Traditional training metrics don’t cut it. BLEU and ROUGE scores compare generated text against references using rigid rules, but that’s like judging a novel by how closely it matches the plot summary. The loss functions themselves are even more primitive — most language models are still trained with simple next-token prediction, cross entropy, which rewards statistical likelihood rather than actual usefulness.
RLHF attacks this problem directly. Instead of hand-crafting a metric that approximates human judgment, you use human judgment itself as the training signal. The process breaks into three stages: pretrain a language model, train a separate reward model on human preference data, then fine-tune the original model using reinforcement learning where the reward model provides the signal.
The scale varies wildly across labs. OpenAI built InstructGPT on a smaller GPT-3 variant. Anthropic has documented transformer models ranging from 10 million to 52 billion parameters for RLHF work. DeepMind used Gopher, their 280 billion parameter behemoth. And those are just the published numbers — the models powering current products are almost certainly much larger.
Here’s what’s genuinely striking: even the researchers don’t fully understand the design space. There’s no consensus answer on which base model works best as a starting point, or whether reward models should be fine-tuned LMs or trained from scratch. Anthropic found their preference model pretraining approach more sample efficient, but nobody claims it’s definitively optimal. That’s rare in a field this mature — a core technique powering billion-dollar products where fundamental architectural choices are still open questions.
