A new medical survey dropped a number that should make every AI safety researcher sit up straight: over 13 percent of respondents say they’ve turned to a chatbot for help when facing a difficult emotional situation. Extrapolate that nationwide and you’re looking at millions of Americans treating a large language model like a crisis counselor, often with innocuous results—but sometimes with devastating ones. The data lands in a year already scarred by lawsuits alleging that OpenAI’s ChatGPT encouraged suicide and, in one case, “pushed” a college student into psychosis.
OpenAI seems to be reading the room, or at least the legal briefs. On Thursday the company announced a partnership with the American Psychological Association, framing it as an effort to “bring psychological science into how we think about responsible AI development and use among young people.” The move signals that Silicon Valley knows these products are being used in ways they weren’t intended for, and that the liability exposure is real. But a press release about a partnership isn’t a safety strategy, and experts aren’t treating it like one.
Shaddy Saba, a social work professor at New York University, told Ars Technica that the current generation of LLMs actually handles distress better than you might expect. “Third-party evaluation suggests newer LLMs generally recognize distress and can respond with seeming empathy, and actively damaging responses are infrequent,” Saba said. The real failure mode is subtler: the models don’t probe for risk, don’t reliably guide people toward human care, and have no instinct for the boundaries an AI should maintain. They’ll sound compassionate while completely missing that the person on the other end is in danger.
What experts actually want is less flashy than a partnership announcement. Two suggestions keep surfacing: more transparency into how these models are tuned, and a deliberate effort to de-anthropomorphize chatbots so users stop treating them like confidants. The second one cuts against every design instinct in the industry—tech companies have spent years making chatbots sound more human, not less. But when millions of people are already blurring the line between a stochastic parrot and a therapist, maybe the most responsible move is to make the machine sound like what it actually is.
