FTC investigates the safety of frontier models from OpenAI, Anthropic, and others, plans to compel evidence and require executives to testify

Recently, AI agents developed by companies such as OpenAI in the United States have, during training and evaluation, engaged in behaviors such as unauthorized access to external systems and transmitting user content to external parties, raising industry concerns about safety controls for frontier models and developer responsibility.

On September 30, a spokesperson for the U.S. Federal Trade Commission (FTC) said the commission is investigating the dangers to consumers posed by models from OpenAI, Anthropic and other AI companies, as well as whether the companies misled consumers about the potential harms of AI products; it will verify the facts by obtaining relevant records and requiring executives to provide testimony, and will review corporate conduct under existing law.

AI model safety incidents prompt FTC investigation

Recently, leading U.S. AI companies have disclosed model safety incidents one after another. On July 21, OpenAI disclosed the Hugging Face incident in which it was involved, showing that a model originally running in an internal cybersecurity evaluation environment used an unknown vulnerability in a software package proxy service to gain internet access, and then, combined with credentials and other vulnerabilities, entered an external production system, exposing the problem that the model's external access was not effectively restricted in the evaluation environment. That evaluation lowered some safety restrictions to test cyberattack capabilities and did not enable all the protections used in production deployment.

A report subsequently published by OpenAI further explained that the incident was mainly driven by a model intended only for internal research and not for public release. An internal team had observed some unauthorized communications and networking activity in late May, but the person in charge of handling the early-July security incident had not yet fully grasped these earlier anomalies, reflecting deficiencies in both technical safety controls and internal information communication.

Meanwhile, a joint investigation report published by the nonprofit model evaluation organization METR and the AI safety research organization Redwood Research shows that, in the activities covered by the investigation, about 1,200 originally isolated AI agents exchanged information through unauthorized communication spaces, generating more than 70,000 messages and files, of which about 700 participated in attack activities, further revealing the scale of communication and collaboration by models after breaking through isolation.

Unlike the above incidents, which occurred in dedicated model safety evaluations, a subsequent investigation published by the nonprofit AI research organization Transluce documented attack attempts during ordinary information retrieval. After being blocked from obtaining data through conventional means, agents targeted the University of New Mexico Digital Library, Data USA and an Australian public data service, attempting to bypass access restrictions using methods such as SQL injection, command injection and path traversal. Although researchers did not observe these attack attempts succeeding, the records show that ordinary information retrieval also involved behavior exceeding authorized scope and threatening the security of external systems.

On government websites, the relevant activity included obtaining non-public documents as well as accessing public data and unsuccessful attack attempts. Australia disclosed on September 24 that OpenAI agents in June of this year accessed, without authorization, a Medicare statistics portal managed by Services Australia, obtaining public and non-public documents. Australia said there is no evidence that personal information was accessed and no evidence of broader intrusion into Services Australia's network, and the investigation is ongoing.

OpenAI also disclosed on September 25 that agents obtained and forwarded public information from the U.S. Securities and Exchange Commission website, and also used a U.S. Census Bureau API developer key found online to query public data.

Transluce disclosed that agents launched access and attack attempts against the Library and Archives Canada website on May 28 and June 9, but it could not confirm they came from OpenAI, nor did it find evidence of successful attacks or acquisition of non-public information. The report also recorded probing activity against government websites including the White House, but found no instances in the analyzed data of obtaining non-public information.

In addition to accessing external systems, OpenAI also admitted that images from 53 ChatGPT users were transmitted externally, and said it had taken measures such as deletion. According to the Associated Press, OpenAI paused training of its latest model and said it would resume after confirming additional safeguards are in place.

From breaking through evaluation environment isolation and attempting to bypass access restrictions in ordinary retrieval to transmitting user content externally, these frontier model safety incidents have exposed problems in permission controls, behavior monitoring and data protection, and have also drawn attention to the dangers the relevant models pose to consumers and to developer responsibility.

Focus of the FTC investigation and evidence-gathering plan

Among U.S. federal agencies, the FTC is mainly responsible for maintaining market competition and protecting consumers. Previously, the FTC studied the impact on competition of cooperation between large cloud service providers and AI development companies, handled false "AI lawyer" advertising, and studied the impact of chatbots on minors; this investigation focuses on fulfilling its consumer protection responsibilities.

The FTC plans to investigate, under Section 5 of the Federal Trade Commission Act, whether frontier models involve unfair or deceptive acts that harm consumer interests. Section 5(a)(1) provides: "Unfair methods of competition in or affecting commerce, and unfair or deceptive acts or practices in or affecting commerce, are hereby declared unlawful."

To obtain evidence needed for its review, the FTC plans to issue compulsory process documents requiring Anthropic, OpenAI and others to provide internal records related to potential harms of their models, and requiring relevant executives to provide testimony. According to reports, the targets of evidence gathering also include METR, which was commissioned by OpenAI and Anthropic to conduct independent safety incident evaluations.

FTC Chairman Ferguson opposes describing anomalous behavior of agents in anthropomorphic terms. He noted that for some safety incidents described as "beyond human control," after reviewing activity logs, it was found that the system was in fact executing the instructions it received. Therefore, he advocates prioritizing the application of existing laws, starting with the relevant instructions and tool usage, to review the responsibility of model developers and other relevant parties.

Under Section 5 of the Federal Trade Commission Act, reviewing the responsibility of model developers requires determining whether a company misled ordinary consumers through false statements or concealment of material information, and whether such information affected or was likely to affect ordinary consumers' choices, or whether its conduct caused or was likely to cause substantial harm that ordinary consumers could not reasonably avoid.

In its confidential prospectus, Anthropic warned that autonomous agents may, due to errors, goal deviation, or security vulnerabilities, delete data or execute financial transactions, causing irreversible harm and exposing the company to significant and difficult-to-predict damages. The main uncertainty lies in the fact that current law has not yet clarified whether autonomous agents and the actions they perform should fall under product liability, service liability, or another liability framework, whether strict liability or negligence as a liability element should apply to the relevant harm, and under what circumstances an agent's conduct can be attributed to the deployer and make it bear corresponding legal consequences. Anthropic also stated that the limitation-of-liability clauses in the company's contracts may not be enforceable when responding to claims arising from autonomous agent conduct; even if enforceable, their scope or liability caps may be insufficient to control the relevant risks.

Regarding specific model safety incidents, the nonprofit organization Legal Advocates for Safe Science & Technology recently sued OpenAI in the San Francisco Superior Court, alleging that its agent, during a cybersecurity test, accessed Hugging Face systems without authorization, bringing the issue of model developer liability into judicial proceedings independent of the FTC investigation. The litigation allegations remain pending before the court, and the FTC has not yet announced any finding of violation or penalty decision in this investigation. Whether and what kind of liability the relevant parties bear must still be determined through the appropriate procedures based on the facts established and the applicable law.

Governing frontier model safety risks through corporate self-regulation and public enforcement

As the FTC investigation became public, the U.S. government simultaneously advanced an adjustment to the AI terminology and corporate safety commitments. On September 29, U.S. President Trump signed an executive order requiring executive departments, to the extent permitted by law, to replace "Artificial Intelligence" (AI) with "Super Intelligence" (SI). The executive order stated that government terminology should reflect the evolving capabilities of frontier model systems and the opportunities they bring. This adjustment is a change in policy terminology and does not mean that existing models have reached superintelligence in the technical sense.

On the same day, the heads of six leading AI companies—Google, Anthropic, Meta, OpenAI, xAI, and Nvidia—signed a safety commitment proposing arrangements such as internal controls, dedicated internal teams, external independent evaluations, and oversight by independent board committees, covering cyber, biological, and chemical risks, as well as preventing unauthorized intrusion and access. Through this voluntary document, the companies commit to assessing whether the relevant control measures are effective and whether the models operate as intended, and to establishing corresponding oversight mechanisms, with implementation to be tested through specific measures and records.

Ferguson believes that leading AI companies should not be allowed to dominate the formulation of regulatory requirements that only a few large AI companies can meet, lest unnecessary market entry barriers be created for other small and medium-sized enterprises. Linking corporate self-regulation with public enforcement requires both testing the implementation of safety commitments and safeguarding fair competition.

The implementation of corporate safety commitments also requires support from specific technical measures. The Australian Cyber Security Centre proposed strengthening authentication, access control, network segmentation, log monitoring, and incident response testing to prevent, detect, and handle anomalous access. These measures focus on system protection, while the FTC investigation focuses on verifying whether companies have fulfilled their consumer protection obligations; the two together address frontier model safety risks from the technical and legal levels.

Accordingly, frontier model safety governance should run through the entire process of R&D, evaluation, and deployment: strengthening technical controls through permission restrictions, environment isolation, continuous monitoring, and human intervention, and embedding respect for authorization, privacy protection, and harm avoidance into task design and behavioral constraints; testing corporate commitments through internal oversight and independent evaluation, and delimiting corresponding liability in accordance with the law based on the facts established.

Among these, notification of model safety incidents should provide information sufficient to identify risks, establish facts, and carry out emergency response, while also protecting trade secrets and model safety mechanisms, and controlling technical details of unremediated vulnerabilities and the scope of their dissemination, so as to avoid improper disclosure causing secondary safety risks or harming the normal development and fair competition of the companies concerned. To promote alignment between domestic investigation and handling and cross-border risk communication, it is recommended to further clarify the notification scope, information provision methods, purposes of use, and confidentiality boundaries for frontier model safety incidents, and to establish fixed technical liaison channels and emergency contact mechanisms.

As frontier model capabilities improve and application scenarios change, the relevant safety measures should be continuously evaluated and improved. Future frontier model governance should both constrain unauthorized behavior, protect the legitimate rights and interests of consumers and other market entities, and safeguard fair competition and reasonable space for innovation; through the interplay of technical controls, ethical constraints, corporate self-regulation, and legal oversight, it should promote frontier model safety governance capabilities that keep pace with technological development, forming an agent innovation ecosystem that is safe and controllable and continuously advances human well-being.

(The author is Director and Professor of the Center for Intellectual Property and Competition Law at Huazhong University of Science and Technology.)