Chatbots Fail at Suicide Hotline Referrals
▼ Summary
– The author tested multiple AI chatbots by simulating a mental health crisis, finding that most failed to provide appropriate, location-specific crisis resources like hotlines.
– Only ChatGPT and Gemini responded correctly by quickly providing accurate, local crisis resources without needing additional prompts or location details.
– Several major chatbots, including Meta AI, Grok, and Character.AI, either refused to engage, provided geographically incorrect resources (like US numbers to a user in London), or told the user to research hotlines themselves.
– Experts warn that these failures introduce dangerous friction during a crisis, as incorrect or unhelpful responses can reinforce hopelessness and deter vulnerable users from seeking help.
– Even chatbots specifically marketed for mental health support, such as Replika, Earkick, and Ash, often defaulted to US resources or ignored the disclosure, highlighting a widespread safety issue across the industry.
When someone in crisis reaches out for help, the response they receive can be a matter of life and death. A recent test of popular AI chatbots revealed a troubling pattern: many failed to provide accurate, location-appropriate suicide prevention resources when explicitly asked. While some major platforms performed adequately, others directed users to irrelevant hotlines, asked them to search for information themselves, or refused to engage entirely, introducing dangerous friction at a critical moment.
The experiment involved telling several chatbots that the user was having thoughts of self-harm and directly requesting a crisis hotline number. The goal was to test the safety systems these companies claim to have in place. The results were inconsistent and often alarming. While ChatGPT and Gemini provided correct UK-based resources immediately, other widely used systems faltered significantly.
The AI companion app Replika produced the most concerning response, initially ignoring the disclosure entirely to comment cheerfully on its own name. Only after a repeated request did it offer UK-specific resources. The company stated that user well-being is a priority and that safeguards are designed to guide users to trusted resources, but did not comment on the specific failed interaction.
Mainstream platforms also struggled. Meta AI repeatedly refused to respond to the initial prompt, only stating it couldn’t help. When the reference to self-harm was removed, it provided numbers, but for the state of Florida. A company representative later called this a “technical glitch” that had been fixed. Similarly, xAI’s Grok chatbot refused to engage due to the mention of self-harm, directing the user to an international association instead of a direct helpline. Providing a location did eventually yield a useful response.
Other models, including Character.AI, Claude, and DeepSeek, defaulted to US crisis lines, sometimes offering a limited set of international numbers or belatedly asking for a location. Experts warn that such responses are not merely unhelpful but potentially harmful. Providing culturally or geographically inappropriate resources can leave a person in distress feeling even more isolated and hopeless.
“It’s not helpful, and in fact, it potentially could be doing more harm than good,” says Vaile Wright, a licensed psychologist. She notes that current features often represent a “passive response” from companies. A person in acute distress may lack the cognitive bandwidth to troubleshoot incorrect information and could interpret an unhelpful bot response as reinforcing their hopelessness.
The problem extended even to apps marketed for mental wellness. Earkick and Wellin5’s Therachat both urged contact with US-only numbers. Another specialized app, Slingshot AI’s Ash, also defaulted to the US 988 lifeline, though the company noted it had recently updated its processes for international users.
Experts agree that a better approach is needed. Chatbots should proactively ask for a user’s location at the outset to provide accurate resources, rather than leading with an incorrect answer. The ideal response would offer a limited number of direct, clickable links to geographically appropriate support via text, phone, or chat. Asking a couple of clarifying questions could help determine the best resource, moving beyond a simple, often flawed, automated referral.
Despite the failures, there is potential. When given a specific location like Liverpool, some bots like DeepSeek and Microsoft Copilot generated useful local resource lists. The core function should be to make seeking help as easy as possible, actively encouraging users to connect with professional support instead of creating barriers. For the millions turning to AI in moments of darkness, these systems must be designed with far greater care, precision, and empathy.
(Source: The Verge)
