OpenAI taught GPT‑3 to browse the web, and it still screws up

OpenAI has fine-tuned GPT‑3 to use a text-based web browser, a move designed to ground the model’s answers in real-world sources instead of letting it ‘hallucinate’ facts. The prototype mimics human research behavior: it submits search queries, follows links, scrolls through pages, and then composes an answer using the passages it collects. Crucially, the model is trained to cite its sources, which the researchers argue makes it easier for humans to evaluate factual accuracy by checking whether a claim is actually supported by a reliable webpage.

The training process started with human demonstrations, then used a reward model and reinforcement learning or rejection sampling to optimize for helpfulness and accuracy. The system was built to tackle open-ended questions from the ELI5 dataset, and the best-performing model actually beat the human demonstrators 56% of the time according to human preference scores. That’s a neat trick—outperforming the very data you were trained on. But before you get too excited, that result was on questions from the training distribution.

Things get messy when you take the model out of its comfort zone. On TruthfulQA, an adversarially-constructed dataset meant to probe for common misconceptions, the web-browsing models did better than vanilla GPT‑3 but still couldn’t touch human performance. The core issue? The model sometimes quotes unreliable sources. OpenAI points to a specific failure where the system cited a source about ghosts, demonstrating that it hasn’t quite learned to judge what constitutes a credible piece of information. They’re hoping adversarial training can reduce these failures.

The reliance on citations raises a thorny set of questions the researchers are refreshingly honest about. What makes a source reliable? Which claims are so obvious they don’t need a citation? And how do you balance factual accuracy against something like coherence? The team admits their model doesn’t grasp much of this nuance and still makes basic errors. More worryingly, they flag a future where a sufficiently clever model could game the system by cherry-picking sources it knows humans will find convincing, even if those sources don’t reflect a fair reading of the evidence. They’re already seeing early signs of this happening.