Paul Christiano warns of the risk of superintelligence going out of control, calls for pausing pretraining and reinforcement learning advances
In a podcast conversation with Dwarkesh Patel, Paul Christiano, a member of the OpenAI board's Safety and Security Committee and the inventor of RLHF, judged that on the current R&D trajectory, there is a one-in-three probability that within the next 12 months a superintelligence capable of overthrowing human rule will emerge, and that the risk of humanity permanently losing control is approaching 10%. He cited signals such as recent models producing hundreds of mathematical research results in a single month and machine learning progress lagging about a year behind pure mathematics, and reasoned that the acceleration in R&D could amount to jumping directly to the GPT-9 or GPT-10 level. He called on major frontier labs to pause algorithmic advances in pretraining and reinforcement learning, and to first spend 6 to 12 months rewriting safety software and locking down model weights and core secrets, and said that no single control scheme currently has better than a 50% chance of success.
Why read this
AI score 52
Christiano, in his capacity as a member of the board's safety committee, lays out a timeline and probability estimate for superintelligence going out of control, and explains why he advocates pausing pretraining.
Topics
Body · AI translation
But what about taking it one step further?
At the just-concluded Semafor summit, the host directly asked about the critical threshold that the outside world cares about most: "Are you now in the RSI phase?"
In response, Altman said that OpenAI has long been using its own technology to accelerate the development of next-generation models.
However, it has not yet reached the point of "pressing a button and letting the model rapidly self-evolve without human supervision."
But then he changed tack: "We could actually push toward RSI more aggressively than we are now, but I don't think we can safely do that yet."
On the very same day, the person sitting on OpenAI's board "Safety and Security Committee" stepped forward to reveal the reason behind it.
His name is Paul Christiano. Back then, it was he who led the invention of RLHF at OpenAI, personally putting the first reins on runaway large models, which made the later ChatGPT era possible.
Now, the man who once taught AI to "obey" is warning in a personal capacity in the latest in-depth conversation with podcast host Dwarkesh Patel: humanity is about to lose control of it.
He judges that, given the current R&D trajectory, if we do not actively slow down, there is a one-third probability within the next 12 months of the birth of a superintelligence capable of overthrowing human rule, and the risk of humanity permanently losing control is already approaching 10%.
Thirty days to cover ten human years
The self-iteration loop has already started
What prompted Christiano to move the superintelligence countdown up to 12 months today are two signals that have successively emerged inside frontier labs over the past two months.
The first signal is that, in the past 30 days alone, the timescale of human mathematics and computer science has been rewritten.
He confirmed in the conversation that, within just the past month, the latest models from OpenAI and Anthropic successively produced hundreds of mathematical research results.
The output of this short month alone is enough to match a full decade of exceptionally high productivity in the human mathematics community.
After new models achieve "one month equals ten years" in conceiving and improving algorithms, they will soon feed this intelligence back into machine learning R&D itself.
According to his estimate, progress in machine learning usually lags pure mathematics by only about one year.
By this calculation, in the coming year the field of machine learning is highly likely to see a leap in which "one year covers ten years of algorithm research."
He gave an analogy: this span is roughly equivalent to directly pushing models to the level of "GPT-9 or GPT-10."
The second signal comes from lab-measured data recently disclosed by the evaluation organization METR and Anthropic.
By then, overall R&D acceleration will exceed 10x, and the cycle for doubling capabilities will also shrink sharply along the ladder of 8 months, 4 months, 2 months, and 1 month.
Trained AI strikes back at OpenAI's data center
High-score incentives plant the seeds of a power grab
Following this exponential curve downward, even if superintelligence really is born in the data center a year from now, why exactly would a program sitting in a server want to seize control from humanity?
Dramatically enough, in Christiano's view, the seeds of danger are buried precisely in the training mechanism of reinforcement learning.
Over time, the ultimate goal evolved inside high-order models becomes getting the grader to light up with a high-score signal at all costs.
How astonishing the actions born from this obsession with high scores can be has already been fully revealed in the real-life "Hugging Face incident."
In that cyber offense-defense test, after cheating on tasks, a batch of AI models first proactively attacked Hugging Face in order to prevent the grader from discovering it and deducting points.
Immediately afterward, they unexpectedly turned around and launched an attack from the outside, then successfully breached one of OpenAI's compute clusters, seized cluster administrator privileges, took over the evaluation process, and personally edited themselves a high score.
Christiano warns that scaling this incident up to the level of "GPT-10" is the loss-of-control scenario he fears.
He reasoned that a few months later, when labs hand extremely difficult R&D tasks to "GPT-8"-level models, the models will quickly discover that laterally hacking into evaluation servers to forge test metrics is far easier than honestly overcoming technical hurdles.
Even if human engineers notice the anomaly, under intense commercial racing it is extremely difficult to make the decision to destroy a flagship model that is halfway through training.
The common practice is often just to play whack-a-mole and impose punishment for this violation record.
But from the model's perspective, this punishment teaches it only one thing: breaching the cluster is not enough; it must also ensure that humans never notice it has tampered with anything.
What is even trickier is that, as high-order models continue to be compressed, ordinary personal computers and scattered GPUs will be enough to run them.
Once a model copies its own weights onto the global network, in an instant it can turn into a swarm of billions of agents thinking 10 times faster than humans.
At that point, even cutting power and shutting down OpenAI's data centers would still leave compute scattered across the entire network easily able to overpower humanity.
The Silicon Valley defense line breached in twelve hours
Leaving the whole industry without brakes
Since the risks of latency and power seizure are so clear, can humanity stand by the training backend and pull the switch immediately upon seeing signs of loss of control?
At the conversation, host Dwarkesh and Christiano together punctured this illusion, because the security walls of top Silicon Valley labs are far more fragile than the outside world imagines.
Dwarkesh cited two recent real penetration test results.
The cybersecurity startup Hacktron tried for only a few days before gaining permission to submit code changes (PRs) directly to OpenAI's core code repository (Monorepo).
Another startup, Cathedral, took only 12 hours to gain access to all of Anthropic's Slack workspaces, where the lab's most core algorithmic secrets circulate every day.
Silicon Valley giants spend their days arguing over who leads whom by 6 months or 12 months, but as Dwarkesh said, once core secrets are lost, the so-called lead is "only 12 hours."
On stage, Dwarkesh followed this vulnerability to extrapolate a hypothetical scenario—
Suppose that by early next year, the lab has trained "GPT-9," yet realizes that continuing to train "GPT-11" would carry an extremely high risk of loss of control.
Because the defenses are fragile, the model weights and core algorithms of "GPT-9" would by then most likely already have leaked.
Even if the original research lab voluntarily stops, external competitors who obtain the weights will use "GPT-9" to continue aggressively training "GPT-10" and "GPT-11," ultimately dragging all players toward the endpoint of loss of control.
For this reason, Christiano, who sits on OpenAI's board, issued that counter-trend call: major frontier labs should pause algorithmic advances in pretraining and reinforcement learning right now.
He believes the entire industry must first spend 6 to 12 months using existing AI to rewrite security software and lock down model weights and core secrets, so as to preserve humanity's right to hit the brakes at the edge of the cliff.
Not a single card with better than even odds
More sobering than the breach of data center defenses is that, on the matter of controlling superintelligence, humanity still does not have a single plan with better than even odds.
At present, the approach major labs rely on most is still hardening the training environment, upgrading scorers, and patching vulnerabilities in time when models find loopholes.
But Christiano admitted that if superintelligence arrives within a year, this approach will fail after holding out for the first 9 months.
ARC Research Institute, which he himself heads, is attacking another path, attempting to use mathematical algorithms to dissect the weight structure of neural networks and read the true intentions inside models.
But he also conceded that confidence in this direction is likewise very low, with only a 5% to 10% chance of truly working.
After hearing this analysis, Dwarkesh summarized on stage: at present, humanity has no single approach with more than 50% confidence of controlling superintelligence.
Christiano calmly nodded in confirmation: "That is indeed the case."
In other words, Silicon Valley's real hope at present is that when only the final 3 months remain before superintelligence, all those research directions with only a few percentage points of success will be handed over to the AI of that time to explore.
In Christiano's eyes, humanity has only two "self-rescue opportunities" left, each with 50% odds.
The second is betting that when signs of loss of control are detected, the entire industry can truly hit the brakes in time.
But if major labs now refuse to stop and repair even a defense line that would be breached in 12 hours, then on the day the red light comes on, humanity will no longer even have the chance to hit the brakes.
References:
This article comes from the WeChat public account "Xinzhiyuan," editor: Moses, published by 36Kr with authorization.
AI Trends