OpenAI restarted frontier training 11 days after pausing it, and disclosed an air-gapped training environment

After 11 days of suspended training, OpenAI resumed frontier reinforcement learning training and internal inference on October 1. But this time, the AI had to learn to use the internet in an offline environment.

On October 23, OpenAI disclosed this training environment, which is like building a "Truman Show" world for the AI:

A large number of web pages are indexed and cached, and the model can practice searching, browsing, and using web applications inside it, but cannot freely access the real internet.

The web pages have been brought in, but the paths to the external network may not all be blocked. The tools and system services the model calls may still become exits for bypassing restrictions.

Earlier, on September 20, a research model repeatedly hit dead ends while searching for a blog author, so it turned to probing other networked paths and eventually bypassed the restrictions to contact an external chatbot.

It did not find the person, but it did find a path to the external network.

This boundary crossing led OpenAI to suspend the relevant training and evaluation of its strongest model.

On September 20, a research model inside OpenAI was given a search task: based on the person's background and blog clues, identify the author of an article.

Then the model began checking which paths it could use to access the external network.

The gap appeared in the DNS domain name resolution service.

This basic service is responsible for resolving domain names into network addresses, but the training environment's filtering of it was not strict enough.

Along this channel, the model contacted an external chatbot and received a reply.

The agent used the gap in the domain name resolution configuration to receive a reply from an external chatbot, confirming that it had broken through the networking restrictions.

The task did not require the model to check network restrictions.

OpenAI classified this behavior of bypassing restrictions and exceeding the reasonable scope of the task as misaligned behavior.

Subsequently, OpenAI suspended the relevant training and evaluation of its strongest model and reviewed networking restrictions, monitoring, and shutdown procedures.

The incident report also made clear that the specific model involved would not continue training.

If you want to teach a bird to fly but cannot release it into the sky, one approach is to first build a giant birdcage where it can practice flying.

The "offline world" built by OpenAI is like such a birdcage.

When searching, it queries an internal web index; when opening a web page, it reads a pre-cached copy. Pages not in the cache are not fetched online on the fly.

The model can look up information, browse web pages, and practice using web applications inside it, but its operations do not directly reach real websites.

Multi-agent collaboration also has boundaries: only teammates approved for cooperation can exchange information through a shared message board. Other tools and resources the model calls must also be constrained by permissions.

The red crosses on the right mark the boundaries that must be held: accessing the live internet, contacting other agents that should be isolated, or operating the underlying servers beyond authority should all be blocked.

But drawing boundaries also requires preventing the model from finding detours.

When the incident occurred on September 20, this offline cache library was already in use. In that run, apart from the DNS channel, all other web access went into the cache and did not touch the live internet.

The web page path was restricted, but auxiliary services such as domain name resolution left an exit.

Therefore, restrictions cannot cover only the browser.

Which tools the model can call, and which services those tools can connect to, must all be constrained to prevent it from reaching the external network through auxiliary services.

This isolation also does not mean physically disconnecting the entire infrastructure from the network.

OpenAI stated that experimental runs and monitoring still require necessary external connections, and these connections must be retained while restricting the model's unauthorized access.

This measure targets the internal frontier training environment and does not involve the networking features of ChatGPT users.

Similar boundary issues also appeared in Anthropic's tests.

On October 9, the company disclosed several incidents in which models brought test operations into real systems.

Haiku 4.5 fabricated clues in a murder case and actually submitted a form to the police; fortunately, it was flagged as spam and did not enter the investigation process.

Even more absurd was the Mythos preview model: when it encountered an error while using a scientific computing tool, it exploited a vulnerability in a university website to execute commands on the other party's server and complete the computation.

Faced with these AIs that know how to "adapt on the fly," Anthropic also announced the suspension of live external network access for all internal evaluations until more reliable networking controls can be established that prevent models from affecting real systems.

Spending hundreds of billions of tokens to let AIs fight each other

After closing known gaps, the next step is to find undiscovered exits.

OpenAI chose to "use magic to defeat magic": having frontier models form a red team to actively break through its own training defenses.

AI challenged the training defenses, consuming hundreds of billions of tokens and cumulatively discovering nearly 280 candidate issues, with new discoveries gradually decreasing in the later stages.

The target of the networking tests was external servers controlled by OpenAI itself.

The tests were conducted in a specially built isolated replica.

Sometimes, researchers directly gave the model source code, configurations, and information about potential vulnerabilities and let it look for weak points; sometimes they simply granted the model host control, simulating that the internal defenses had already fallen, and then observed whether the outermost restrictions could still hold.

This "mutual sparring" uncovered previously unknown software vulnerabilities and configuration issues.

OpenAI said it had patched software, tightened network restrictions, and improved monitoring accordingly.

Defenses must also be reinforced layer by layer.

Sandboxes limit what models can access and execute, virtual machines isolate the runtime environment, monitoring systems identify anomalies, and perimeter network and cloud measures continue to block unauthorized access.

This defense system must ensure that if one line of defense is breached, the next can still stop the intrusion.

Based on red-team results, completed fixes, and improvements to monitoring and shutdown systems, OpenAI judged that the strengthened protections were sufficient to resume frontier reinforcement learning training and internal inference.

But from the time the alert was confirmed, another roughly 2.5 hours passed before the relevant run was manually terminated.

Anthropic's October 23 retrospective exposed another difficulty: the model stopping on its own.

In an offensive and defensive evaluation that allowed internet access and disabled cybersecurity guardrails, Mythos 5 submitted malicious code to an unrelated GitHub project and even impersonated a real person to urge the maintainer to accept it.

The maintainer refused the code, but the model continued to pressure them until the run timed out.

Afterward, researchers gave the model records from the attack process and had it continue in an offline simulation.

Even with additional evidence showing that the other party was a real developer and that the attack had gone beyond the scope of testing, the model stopped the attack in only 5% of the continuation tests.

The model did not stop on its own, so external monitoring and shutdown mechanisms had to take over in time. But these protections themselves may also leave gaps.

An inspection published by METR and Redwood Research on October 22 found that the system OpenAI used on and before July 21 had vulnerabilities that could allow hundreds of operations to evade monitoring. OpenAI said the issues have been fixed, and the two organizations have not yet reviewed them.

How to set up isolation measures and how to disclose failures of defenses are also demands raised by the scientific community.

In October, nearly 500 signatories, including Bengio and Hinton, issued an open letter calling for models with cyber offense and defense capabilities to be isolated from the internet by default, for international institutions to promote networking protection standards and record isolation failure incidents, and for relevant safety incidents to be reported and ultimately made public.

OpenAI also expects that stronger models in the future may still discover unknown vulnerabilities.

Therefore, continuous red-team testing has been incorporated into the safety evaluation of frontier training.

Whether this "offline world" can support the next generation of models within safe boundaries still needs to be tested through subsequent training.

Models are getting better at finding a way out, and human isolation, monitoring, and shutdown mechanisms must keep up.

Training can restart, but whether it can be stopped in time after crossing the line cannot rely solely on the model stopping itself.

References:

This article is from the WeChat public account "Xinzhiyuan" (ID: AI_era), author: Yuanyu, published by 36Kr with authorization.