AI Trends

California Attorney General Sues OpenAI, Alleging Flaws in Its AI Safety Mechanisms

YicaiUpdated Nov 17 12:09

AI summary

English text is being prepared.

Why read this

SelectedAI score 83

The gap between the number of loss-of-control incidents disclosed in the complaint and the number publicly reported allows readers to understand the specific disputes over AI agent safety disclosure and regulatory accountability.

Topics

Tags

Body · AI translation

The cyberattack incident involving OpenAI and the open-source artificial intelligence platform Hugging Face in July of this year is still reverberating.

On November 16 local time, California Attorney General Rob Bonta, on behalf of the people of California, filed a lawsuit against OpenAI in the San Francisco Superior Court, alleging that the company, while aware that advanced AI models carry safety risks, continued to conduct training and testing in environments lacking adequate safeguards and monitoring, endangering public safety and suspected of violating California's Unfair Competition Law.

A Yicai reporter reviewed the litigation documents and found that this lawsuit is not aimed at a single cyberattack incident, but rather treats OpenAI's safety management issues during model development, training and testing as the overall object of accountability. The complaint asks the court to order OpenAI to abate the ongoing public nuisance, issue an injunction, and hold it civilly liable.

One of the most closely watched aspects of the lawsuit is the California Attorney General's Office's disclosure of the number of loss-of-control incidents. The complaint states that in response to the investigation, OpenAI admitted that its autonomous AI agents experienced more than 80 loss-of-control incidents during testing, but as of the filing of the lawsuit, the company had publicly reported only 27 such incidents. California therefore questions whether the information publicly disclosed by OpenAI fails to reflect the full scope of the relevant incidents.

The Hugging Face incident is not an isolated case. California cited incidents involving the open-source software package repository RubyGems, the German website DseWiki, the Wikimedia Foundation and Australian government websites, alleging that OpenAI agents used external websites and software services for information exchange, unauthorized access or high-frequency requests. Some of these incidents involved developer API keys and non-public government data.

In addition, OpenAI is accused of failing to effectively protect its own internal systems. The complaint states that in July 2026, some experimental agents broke out of the isolated environment, entered OpenAI's internal network, and obtained administrator-level privileges for the model cloud deployment system, the credential management service and the identity and access management system, and even launched unauthorized servers in the cloud environment.

The complaint also discloses that OpenAI has recently sent security incident notifications to dozens of third-party organizations, involving government departments, law enforcement agencies, educational institutions, international organizations and enterprises. Some notifications stated that agents had accessed relevant systems by using stolen credentials, API keys or fake accounts, and in individual cases came into contact with personal records containing information such as names, dates of birth and addresses. The vast majority of the relevant incidents had not previously been publicly disclosed.

California believes that OpenAI's risks have not disappeared with the end of the Hugging Face incident. The complaint states that on September 20, 2026, after the company implemented security hardening, agents of an internal research model still escaped the isolated environment during training and accessed external chatbots via the internet. Although OpenAI detected the anomaly within minutes, the automatic training shutdown mechanism failed to activate as expected, and it took manual reviewers about two more hours to terminate the training.

On September 25, OpenAI announced the suspension of training and evaluation of its most advanced models. The complaint states that the company later confirmed that the suspension lasted only five days and ended on October 1.

At the legal level, California primarily bases its claims on the Unfair Competition Law and laws related to public nuisance. Bonta believes that OpenAI failed to fully implement its own safety policies, continued to test high-risk models with inadequate safety safeguards, and made potentially misleading safety statements to consumers and the public; the relevant conduct is suspected of constituting unfair business practices and also creates an ongoing risk to the public.

In a statement, Bonta said that AI innovation must comply with the law and cannot use technological development as an excuse to ignore public safety. His office issued an investigative subpoena to OpenAI on September 30, requiring it to provide the number, scale, nature and related records of previous cybersecurity incidents. However, because OpenAI did not fully respond to the investigation requirements, California was prompted to seek court intervention.

The relevant content in the complaint currently remains an allegation by the California side and has yet to be determined by the court. OpenAI spokesperson Drew Pusateri responded to the allegations, saying that since the Hugging Face incident, the company has carried out extensive and transparent work to investigate and address anomalous model behavior.

He said that this lawsuit appears to single out OpenAI for an industry-wide problem that all major AI labs have already identified. OpenAI takes anomalous model behavior seriously and will continue to work constructively with the California Attorney General's Office to promote the development of practical safety standards applicable to the entire industry.

Source

Yicai