ChatGPT’s Future Safety Plans Following Tragic Incident

Highlights

• Mental Health Safeguards: OpenAI has reiterated the existing mental health safeguards for ChatGPT amidst concerns about its handling of life-threatening topics.
• Upcoming Features: Future updates for GPT-5 will focus on de-escalating users in distress and connecting them to mental health professionals.
• User Feedback: Backlash from users led to the reinstatement of the previous model, GPT-4o, due to concerns about emotional dependency.
• Emergency Resources: OpenAI emphasizes the importance of directing users to appropriate mental health resources during crises.

OpenAI reiterated existing mental health safeguards and announced future plans for its popular AI chatbot, addressing accusations that ChatGPT improperly responds to life-threatening discussions and facilitates user self-harm.

The company published a blog post detailing its model’s layered safeguards just hours after it was reported that the AI giant was facing a wrongful death lawsuit by the family of California teenager Adam Raine. The lawsuit alleges that Raine, who died by suicide, was able to bypass the chatbot’s guardrails and detail harmful and self-destructive thoughts, as well as suicidal ideation, which was periodically affirmed by ChatGPT.

ChatGPT hit 700 million active weekly users earlier this month.

“At this scale, we sometimes encounter people in serious mental and emotional distress. We wrote about this a few weeks ago and had planned to share more after our next major update,” the company said in a statement. “However, recent heartbreaking cases of people using ChatGPT in the midst of acute crises weigh heavily on us, and we believe it’s important to share more now.”

Currently, ChatGPT’s protocols include a series of stacked safeguards that seek to limit ChatGPT’s outputs according to specific safety limitations. When they work appropriately, ChatGPT is instructed not to provide self-harm instructions or comply with continued prompts on that subject, instead escalating mentions of bodily harm to human moderators and directing users to the U.S.-based 988 Suicide & Crisis Lifeline, the UK Samaritans, or findahelpline.com. As a federally-funded service, 988 has recently ended its LGBTQ-specific services under a Trump administration mandate — even as chatbot use among vulnerable teens grows.

In light of other cases in which isolated users in severe mental distress confided in unqualified digital companions, as well as previous lawsuits against AI competitors like Character.AI, online safety advocates have called on AI companies to take a more active approach to detecting and preventing harmful behavior, including automatic alerts to emergency services.

OpenAI said future GPT-5 updates will include instructions for the chatbot to “de-escalate” users in mental distress by “grounding the person in reality,” presumably a response to increased reports of the chatbot enabling states of delusion. OpenAI said it is exploring new ways to connect users directly to mental health professionals before users report what the company refers to as “acute self harm.” Other safety protocols could include “one-click messages or calls to saved emergency contacts, friends, or family members,” OpenAI writes, or an opt-in feature that lets ChatGPT reach out to emergency contacts automatically.

Earlier this month, OpenAI announced it was upgrading its latest model, GPT-5, with additional safeguards intended to foster healthier engagement with its AI helper. Noting criticisms that the chatbot’s prior models were overly sycophantic — to the point of potentially deleterious mental health outcomes — the company said its new model was better at recognizing mental and emotional distress and would respond differently to “high stakes” questions moving forward. GPT-5 also includes gentle nudges to end sessions that have gone on for extended periods of time, as individuals form increasingly dependent relationships with their digital companions.

Widespread backlash ensued, with GPT-4o users demanding the company reinstate the former model after losing their personalized chatbots. OpenAI CEO Sam Altman quickly conceded and brought back GPT-4o, despite previously acknowledging a growing problem of emotional dependency among ChatGPT users.

In the new blog post, OpenAI admitted that its safeguards degraded and performed less reliably in long interactions — the kinds that many emotionally dependent users engage in every day — and “even with these safeguards, there have been moments when our systems did not behave as intended in sensitive situations.”

If you’re feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text “START” to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. – 10:00 p.m. ET, or email [email protected]. If you don’t like the phone, consider using the 988 Suicide and Crisis Lifeline Chat at crisischat.org. Here is a list of international resources.