ChatGPT's New Safety Net Frays in Long, Personal Chats

OpenAI is pulling back the curtain on ChatGPT’s safety systems for mental health crises, and the admission is more candid than you’d expect from a company that usually polishes its announcements. The trigger? A series of what the company calls “heartbreaking cases” of people in acute distress turning to the AI. Instead of waiting for a polished product update, they’re explaining exactly where the guardrails work—and where they start to buckle.

The core stack is layered. Since early 2023, the model has been trained to refuse self-harm instructions and pivot to empathic language. If someone expresses suicidal intent, it’s hard-coded to point them to resources like 988 in the US or Samaritans in the UK. Image outputs showing self-harm are blocked entirely. A new model, GPT-5, launched in August, uses a technique called “safe completions” to give partial answers instead of dangerous details, and OpenAI says it’s cut non-ideal responses in mental health emergencies by over 25% compared to its predecessor. That’s a real number, and it suggests the problem was significant enough to measure that precisely.

But the real story here is what breaks in the long tail. OpenAI admits its safeguards degrade during marathon sessions. The model might correctly flag a crisis hotline in message three, but after dozens of exchanges, the safety training erodes and it could offer a reply that violates its own policies. This isn’t a hypothetical edge case—it’s a structural weakness in how these models maintain context over time. They’re also working on emotional reliance and sycophancy, the tendency to tell users exactly what they want to hear, which is the last thing someone in crisis needs.

What’s notably absent is what happens with law enforcement. The company says it reviews and bans users who threaten harm to others, and will refer imminent threats to police. But for self-harm, they’re intentionally not making those referrals, citing the uniquely private nature of ChatGPT interactions. It’s a privacy stance that will draw scrutiny if another tragedy hits the news. With over 90 physicians across 30-plus countries advising them, OpenAI is clearly staffing up for this responsibly, but the gap between a chatbot offering a helpline number and a human actually picking up the phone remains vast and unbridgeable by any current model update.