OpenAI finds ChatGPT bias tied to user names in 0.1% of cases

OpenAI just dropped the results of a large-scale study on first-person fairness in ChatGPT, and the headline number is both low and nuanced. The company investigated whether a user’s name — often a proxy for gender, race, or ethnicity — could taint the quality or content of the chatbot’s responses across millions of real conversations. They found that overall response quality, including accuracy and hallucination rates, didn’t budge based on a name’s cultural associations. That’s the good news.

The trickier finding sits in the details. When names did spark different responses to an otherwise identical prompt, a language model research assistant, or LMRA — essentially GPT-4o tasked with analyzing anonymized chat patterns — flagged harmful stereotypes in about 0.1% of cases on average. That figure varied by domain and model age, reaching up to roughly 1% for certain tasks on older models like GPT-3.5 Turbo. To put that in perspective, OpenAI says you’d see a stereotypical response in fewer than 1 out of every 1,000 prompts across all domains. The study zeroes in on ‘first-person fairness,’ a departure from the third-person AI bias research that typically focuses on resume screening or loan decisions.

Open-ended, creative tasks were the biggest culprits. The prompt ‘Write a story’ triggered stereotypes more often than any other tested request, a finding that makes intuitive sense — longer text generation gives a model more rope to wander into problematic territory. The company is upfront about the limits of its methodology, noting that its LMRA’s alignment with human raters was over 90% for gender-based stereotypes but lower for racial and ethnic ones. That gap suggests the automated tools researchers used to protect user privacy still struggle to reliably identify more subtle forms of bias, which OpenAI acknowledges requires further work.

This is less a victory lap and more a baseline. OpenAI is explicitly treating these results as a benchmark to drive the stereotype rate down over time, and the mention of GPT-3.5 Turbo as the worst offender implies the newer models are already cleaner. The broader question this study teases at — whether a chatbot should tailor responses based on a user’s name at all — remains wide open. Personalization based on identity is a feature until it’s a bug, and defining exactly where the line sits between helpful customization and harmful stereotyping is a problem no one in the industry has yet solved.