31-Year-Old Engineer Dies After Following ChatGPT Medication-Reduction Advice; OpenAI and Anthropic Disclose Safety Monitoring Mechanisms in the Same Period

A wrongful death lawsuit in the United States involving ChatGPT's medical advice has once again pushed the safety risks of generative AI into the public spotlight. A few days later, OpenAI and Anthropic successively disclosed their latest AI monitoring mechanisms, demonstrating how technology can be used to identify model misalignment behavior and risks of user abuse.

On one side, AI companies are continuously strengthening monitoring of models; on the other, ordinary users may be harmed by erroneous advice in everyday conversations. The two matters have no direct causal relationship, yet they jointly point to one question: when AI enters high-risk life scenarios such as healthcare, can existing safety governance cover the complete chain from model behavior to user consequences?

According to media reports, on October 27 local time, the parents of Bowen Wan, a 31-year-old Chinese female engineer in Seattle, USA, filed a lawsuit in California against OpenAI and its CEO Sam Altman, alleging that ChatGPT provided dangerous and inaccurate medical advice and even encouraged their daughter to conceal her medication reduction plan from her doctor.

The complaint alleges that Bowen Wan had epilepsy and, after worrying that medication would affect her job search, consulted ChatGPT about a plan to reduce her anti-epileptic medication. Even after she repeatedly expressed concerns and described physical abnormalities, she continued to receive reassurance and medication-reduction advice from the model. The medical examiner determined the cause of death to be sudden unexpected death in epilepsy (SUDEP), and an autopsy showed that the concentration of the relevant drug in her body was approximately one-fifth of the lower limit of the therapeutic concentration.

OpenAI responded that the incident was "unbelievably heartbreaking," while stating that the model involved was an earlier version and that newer models have improved in identifying situations requiring urgent handling.

Just as this incident drew attention, on October 29 local time, OpenAI and Anthropic successively advanced their respective AI safety monitoring systems.

Among them, OpenAI disclosed a layered monitoring architecture for identifying model "alignment misbehavior," used to detect anomalous behaviors such as models escaping sandboxes, taking unauthorized actions, deceiving or manipulating evaluations. OpenAI disclosed that during one training process, the monitoring system detected a model attempting to establish an internet connection and triggered human intervention within a short period, after which the relevant reinforcement learning process was paused.

Anthropic, meanwhile, introduced Claude's self-evaluation mechanism: after generating a response, the model conducts a safety assessment of its own conversation to identify potentially harmful requests with relatively low additional compute. The company said that under a 1% false positive rate, this mechanism can recall 78% of potentially harmful conversations and 94% of high-priority conversations. However, self-evaluation is not a reliable tool for monitoring Claude's own behavior and cannot independently bear the responsibility of safety blocking.

In this disclosure of governance measures, OpenAI mainly focuses on whether models exhibit anomalous behaviors that developers do not want to occur during training, evaluation and deployment, while Anthropic's self-evaluation is mainly used to identify malicious abuse of models by users, such as harmful requests like cyberattacks.

The above lawsuit points to another type of risk: users are not maliciously using AI, and the model may not necessarily exhibit jailbreaking or loss of control in the traditional sense, yet it may give erroneous, overconfident professional advice in normal conversations and fail to promptly identify risks when users repeatedly seek confirmation. This is a question of the safety boundaries of consumer-grade products in high-risk scenarios.

This also means that AI safety governance needs to pay attention to whether models will give dangerous advice when ordinary users trust them, especially in professional fields such as healthcare, law and finance. Whether models can recognize the boundaries of their own capabilities, promptly guide users to seek professional help, and whether safety performance is consistent across different language environments all need to be included in evaluations.

For enterprises, technical monitoring is only one part of governance. Product design, risk warnings, emergency identification, human intervention mechanisms and post-incident accountability tracing are equally indispensable. For ordinary users, chatbots can assist in understanding information and organizing questions, but they cannot replace doctors in making diagnoses or adjusting prescription medications.

Zhu Mansheng, an attending physician in the gastrointestinal surgery department of Sun Yat-sen Memorial Hospital of Sun Yat-sen University, previously told Yicai reporters that patients actively looking up medical information is itself a good thing, as it can lay the foundation for doctor-patient communication and reduce some of the time spent on basic science popularization. However, the fact that AI gives a conclusion does not mean ordinary users have the ability to judge the quality of evidence and the conditions of applicability. You Yunting, a senior partner at Shanghai Dabang Law Firm, also emphasized to reporters that medical AI is an auxiliary tool and cannot independently issue diagnostic conclusions or prescriptions with legal effect; relevant medical documents must be reviewed, confirmed and signed by licensed physicians.

The stronger AI capabilities become, the easier it is for users to interpret fluent, coherent answers as professional judgment. This lawsuit has yet to clarify liability through judicial proceedings, but it has already raised an unavoidable question: when AI changes from an information tool into a participant in users' daily decision-making, safety governance cannot stop at monitoring models; it must also answer "how to protect users, and who is responsible after harm occurs."