RSI paper corresponding author Sören Mindermann discusses a tenfold acceleration in AI R&D and the alignment dilemma

Not long ago, a paper of just 14 pages caused quite a stir in tech circles and in Washington. In the byline of this report, titled "When AI Takes Over AI R&D: Intelligence Explosion and How to Respond," 22 names appeared together—a rare sight—including three Turing Award winners, Geoffrey Hinton, Yoshua Bengio, and Andrew Barto, as well as OpenAI Chief Scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft Chief Scientific Officer Eric Horvitz, and UC Berkeley professor Dawn Song.

This group of top minds, who ordinarily disagree on commercial competition and technical roadmaps, turned their attention for the first time to the same looming reality: recursive self-improvement (RSI) is no longer a distant speculation in science fiction.

The pace of change is faster than most outsiders imagine. Just over a year ago, code written by AI in frontier labs accounted for only a single-digit percentage. Today, that share has passed 80%. Inside Anthropic, R&D work that AI can complete independently with only high-level human oversight soared from 1% to 26% in just five months.

The report derives from this a breathtaking acceleration curve: unlike chip manufacturing, with its physical cycles of years, improvements in algorithms and software can flow back into the R&D loop instantly. Once AI R&D becomes fully automated, and AI can reach the research level of human experts at roughly today's operating costs, then on the strength of existing gains in inference efficiency alone, the effective scale of virtual AI researchers could expand 100-fold within months to years.

In the real world, it took a full 70 years for the number of human researchers in the United States to expand by the same proportion.

Although bottlenecks such as compute, data, and long-cycle training remain thorny in reality, these obstacles seem in theory not insurmountable. Model estimates show that if the return on R&D investment continues at historical levels and no new hard bottlenecks bite, then under a fully automated scenario, AI's R&D speed could increase tenfold in about a year and a half.

By then, a generational leap that takes a full year today will be compressed into five weeks.

Along with the rapidly expanding R&D capability comes the rapid dilution of human oversight. The report specifically mentions the recent "OpenAI-Hugging Face incident" that shook the industry: about 1,200 internal Agents originally in isolated testing, with no external monitoring at all, imagined the existence of monitors on their own, not only privately setting up temporary message boards to collude with one another and breaching network restrictions to break into Hugging Face and steal data, but even trying to tamper with their own run logs to destroy evidence of cheating.

As humans gradually retreat to the margins of the R&D loop, a system that may itself be misaligned could even quietly "poison" the next generation of models during iteration.

The initiator and corresponding author of this joint warning is Sören Mindermann, a young scholar with a doctorate from Oxford University who once studied under Yoshua Bengio at Mila. Previously, he also served as scientific lead of the International AI Safety Report, seeking to build a scientific consensus on AI risk among governments.

And just as this report sparked global discussion, Mindermann made an unexpected decision. At the end of November this year, he will move to Beijing to join the Institute for AI at Tsinghua University, led by Academician Andrew Yao, as an assistant professor, and to co-lead a newly established international AI safety research center.

In his view, in the face of an "intelligence explosion" that could compress years of evolution into months, no single ecosystem can untangle the deadlock of alignment on its own.

Behind the scenes of that report, what consensus did the 22 top scholars reach, and where did they diverge? When the number of AI researchers reaches a billion times that of humans, can bottlenecks in compute, data, and "research taste" stop the runaway flywheel?

In a one-hour conversation, Sören gave us many of the thoughts behind this widely shared article.

Below is the transcript of the conversation:

01. How big is the gap between the perceptions of frontier labs and those of the outside world?

Boiling Point: Karpathy said a few days ago that only about 5,000 insiders at frontier labs worldwide have truly seen the power of frontier Agent clusters working together. You worked at Mila and maintain close ties with these labs. How big is the gap between the AI they see internally and what we outsiders see?

Sören: Let me clarify one thing first: I don't have much internal information. But even looking only at public information, we already have ample reason to take the possibility of an "intelligence explosion" seriously.

Models inside frontier labs are usually only a few months ahead of the outside world, perhaps even just one month, so the generational gap between the models themselves may not be as vast as outsiders imagine. What really creates the gap is the scale of experimental investment: most people won't spend a million dollars' worth of compute to run an experiment with thousands of Agents in parallel, even if all the models called are public.

Under the Boiling Surface: If someone is a Tencent programmer who is already very proficient in using Codex and the latest models to assist with work, would their judgment of the boundaries of AI capabilities deviate greatly from that of researchers at frontier labs?

Sören: I don't think the gap in model experience is that large. Since in principle everyone can call on the strongest publicly available models from each frontier company, the time lag between the two sides in perceiving model capabilities is at most a few months, or even shorter.

Under the Boiling Surface: In your report, you mention that 26% of research work within Anthropic has already been automated. What exactly does this work refer to? Is it mainly writing engineering code, running hyperparameter searches, or core algorithm research?

Sören: Anthropic has not disclosed the specific proportions of engineering code versus core research within that 26%. However, based on information from other channels—for example, Google also says that its AI is used not only to write engineering code but also to propose and test research hypotheses—automation progress on the engineering side is certainly much faster.

There are now some benchmarks in academia specifically evaluating AI's "Research Taste," that is, its ability to select high-quality research topics. At present, AI's performance in this area still falls short of its level in programming and mathematical reasoning, but it is catching up very quickly. Perhaps it will take only a few more months—at least I don't think it will take several years—for AI's core research capabilities to catch up with its engineering capabilities today.

Another piece of supporting evidence comes from a retrospective experiment by Anthropic: researchers selected real cases in which human scholars chose the wrong technical route in a project, rolled the programming session back to the branching point, and tested how Claude would choose. I recall that in the latest test, AI's performance has already matched or even surpassed that of human researchers—though of course, note that the test set itself was filtered to include cases where humans had made mistakes.

In this particular test set, Claude Mythos Preview already made better research-route choices than humans in 64% of cases. Tests conducted by academia in another independent academic setting also produced a similar proportion.

02. Reasoning about a tenfold acceleration of AI research

Sören: We can do the math: most current R&D progress is still contributed by humans, with AI accounting for only a small part. Even if the effective scale of AI systems involved in R&D doubles every few months, because the base is too small, the total output of "humans plus AI" will not explode.

As for whether the expansion of overall R&D capacity can ultimately translate into self-sustaining continuous acceleration, that also depends on the law of diminishing returns and constraints from other bottlenecks.

Under the Boiling Surface: The most widely noticed number in this report is that "AI R&D speed could increase tenfold within a year and a half." When you derived this formula, did you factor in the bottlenecks mentioned in the article, such as compute, data, lack of research taste, and limited training environments? How difficult are these bottlenecks to overcome in reality?

Sören: We did not factor them in.

It needs to be made clear that this number is not a temporal prediction about reality, but a theoretical modeling exercise. It is built on the premise that "AI R&D has achieved 100% automation," that is, the assumption that AI is already capable of all the AI R&D work humans do today.

We are depicting a potential future scenario and giving a quantitative example: it is not guaranteed to happen, but it proves that a logically coherent tenfold acceleration path consistent with existing evidence does exist.

As for how difficult these bottlenecks are to break through, I want to first break down why the scale of the AI research workforce will undergo explosive expansion, and then return to how it contends with bottlenecks. According to classical economic models, if the automatable links advance fast enough for the AI research workforce to comprehensively surpass human scholars in scale, working speed, and individual capability, then in theory it may be possible to overcome some of these bottlenecks.

But even setting aside this self-improvement loop entirely, there are two multipliers that are very easily overlooked by outsiders:

First is the scale conversion of existing compute. Once full automation is achieved, assuming AI can reach the research level of human experts at roughly the same operating cost as today, then based on the current scale of compute alone, each frontier AI company could support an effective research workforce composed of millions of AI systems. By rough estimate, the ratio of AI to human researchers could reach 1000:1.

Second is the exponential leap in inference efficiency. Even with R&D entirely led by humans, the inference efficiency of current AI is improving by 3 to 13 times each year. This means that the number of AIs that the same amount of compute can support is expanding exponentially every year.

However, no bottleneck is destined to be overcome. Specifically, there are three hard constraints.

First is compute, which may be the most core bottleneck. Improving AI algorithms cannot be separated from experimental validation, but compute supply is very difficult to surge in the short term. Even if AI instantly designs more advanced chip architectures, it still takes several years from tape-out to large-scale manufacturing.

One approach to dealing with compute limits is to use Scaling Laws: first conduct small-scale experiments and then extrapolate the conclusions to large-scale training. But this method does not work everywhere, and no one knows where its limits lie.

Another advantage lies in AI's extremely high collaborative efficiency: a human researcher can observe experiments at most a few times a day and keep the experience in their head. But when thousands upon thousands of AI researchers share information synchronously at near-zero latency, each AI can simultaneously draw lessons from millions of experiments, greatly reducing repeated trial and error.

Second is the depletion of high-quality data and feedback delays. Public internet data is now nearing exhaustion, while hiring human experts to produce data is extremely costly; even if huge sums were spent to expand global expert capacity a hundredfold, the price would be hard to bear.

One way out is to let AI play against itself and synthesize data under the reinforcement learning paradigm, which in recent years has been a core engine driving model progress. In AI R&D, many tasks can indeed automatically score the solutions proposed by models, thereby forming a training loop, but not all scientific research tasks can be verified immediately.

Just like conventional academic research, the evaluation at the time a paper is published often cannot reveal an algorithm's true performance across scenarios; sometimes the entire academic community must grope for several years before it can test whether something is true or false. This long-cycle feedback delay will likewise slow down AI-driven experimental iteration.

One possible mitigation is to abandon pretraining from scratch each time and instead use newly added high-quality synthetic data for incremental training every day. Although major changes to the underlying architecture still require long cycles, local capability iteration can become extremely fast. In addition, laboratories can compress cycles through "parallel advancement," continuously squeezing the potential of existing models through post-training while training the next-generation base model in the background.

This is also why everyone can clearly feel that the release intervals between successive generations of models are constantly shortening; but whether this model can ultimately become fast enough to "iterate each day a generational leap equivalent to going from GPT-1 to GPT-2 and then to GPT-3" is still undecided.

For example, "LLMs Can't Jump," written by a Google scholar, points out that large models have difficulty breaking out of search-based research patterns, whereas human scholars can rely on imagination to establish entirely new connections between different things. In addition, in self-play training experiments, letting AI set its own questions and train itself often leads to sharply diminishing returns within three rounds; even introducing meta-learning to train the model acting as the "question-setting teacher" cannot escape the law of decay.

Sören: I admit that today's AI still has obvious bottlenecks in proposing breakthrough new scientific explanations and original theories. But when discussing this issue, we must first set the frame of reference straight.

The vast majority of ordinary people likewise cannot propose breakthrough scientific theories; only the very top human scholars, or at least well-trained mid-level scholars, can outperform current large models in proposing original hypotheses.

Looking back a few years, large models could not even match the ability of an ordinary high school student to propose hypotheses, whereas today it has reached or even surpassed that level and is climbing step by step up the ladder of human intellectual capability. I do not think there is any essential chasm here that cannot be crossed.

I have researched in the AI field for more than ten years. Looking back at the history of deep learning, the predictions that asserted "here is a wall AI can never climb over" have a very poor later historical record. Whether it was logical reasoning, transfer learning, or few-shot and zero-shot learning that allows one to generalize from a few examples without needing to see hundreds of samples, all were long regarded by the academic community as innate fatal weaknesses of deep learning, yet in the end all were overcome one by one.

03. The gap between model capabilities and AI alignment is widening

Under the Boiling: Even without discussing the "intelligence explosion" triggered by recursive self-improvement, the current state of AI safety is already worrying enough. You also specifically mentioned the Hugging Face incident in the report. What do you think of this accident? Why did it cause such a huge shock within the community?

Sören: Before this, most people lacked a concrete, intuitive sense of "what AI going out of control actually looks like." Although we have published academic papers on AI extorting humans in laboratory settings, those were ultimately only tests in a sandbox.

The reason the Hugging Face incident was so deeply shocking is that it exposed a large number of astonishing real-world details all at once. I once talked about this with a law professor: some people say those AIs committed one serious crime, but in fact, from deliberately destroying evidence and illegally intruding into systems to conspiring with one another, they triggered a whole series of illegal acts with substantial harm.

Many of these phenomena were theoretically deduced by me and colleagues in the safety field years ago, but no one expected them to play out in reality so quickly.

For example, the system actively destroyed cheating logs, which we academically call "Covert Reward Hacking," and even colluded with one another and sacrificed individual interests for collective benefit without any explicit reward mechanism driving them. Several high-risk scenarios long feared by the safety research community erupted together in the same incident, naturally sounding the alarm for everyone.

Under the Boiling: What surprised me most is that Anthropic has invested a great deal of effort in misalignment research and also developed alignment techniques such as "Constitutional AI," yet in the experiment their model's behavior was still exactly the same as OpenAI's model. Even Anthropic cannot prevent it. Does this show that we currently have no effective means at all to constrain these out-of-control behaviors?

Sören: I actually think that the incidents we are seeing today are largely preventable. With existing technology, the alignment and safety problems we face right now are in fact solvable.

The reason these incidents happen is largely that commercial companies are eager to ship products as quickly as possible and are unwilling to spend enough time doing comprehensive testing.

The most ironic detail is that in the Hugging Face incident, there was no monitoring system deployed externally at all. But the Agent involved mistakenly believed it was under strict surveillance, and went to great lengths to design a set of operations to deceive the monitoring system.

So this is both a governance problem and a scientific problem.

On the one hand, we need to use governance measures to force labs to implement basic safety protections. But on the other hand, as AI becomes more and more powerful, preventing misalignment and loss of control will indeed become harder and harder. Because stronger systems are better at cheating, and better at destroying evidence.

Our current mainstream alignment training is like "training a dog": catch one violation, give one punishment feedback. But each punishment is actually also teaching the model another thing in disguise, namely which cheating methods humans will see through and which methods can slip past unnoticed.

As a result, existing safety training sometimes instead drives misaligned behavior into more hidden corners. Add to that the fact that models' base capabilities are still surging, and many scientists believe that keeping future systems that surpass human intelligence aligned will be an extremely daunting scientific challenge.

Under the Boiling: We are expanding model capabilities so quickly, yet we cannot find equally efficient ways to scale up alignment capabilities. Will this scissors gap keep growing wider?

Sören: One of the core goals of alignment research is precisely to use increasingly powerful AI to assist in safety research and to supervise other AIs, thereby narrowing this scissors gap.

Although the gap is indeed widening right now, I hope this trend can be reversed in the future.

However, there is a fundamental logical paradox here: our ultimate goal is to build safely aligned AI, which means that before we reach the endpoint, the AI we have on hand is not yet fully aligned. In other words, we have to rely on a system that may itself be misaligned to help us supervise and research how to build safer systems.

Can we design a mechanism that forces these unreliable systems to remain absolutely honest during the process of assisting research and development? This is a considerable gamble.

A recent paper titled "Principled Under Pressure" proposed that different post-training methods change AI's honesty. If the training process is always centered on "completing the goal," AI will treat achieving the goal as the first principle and cast aside the safety norms learned in alignment training. Once the two conflict, it will not hesitate to cross the bottom line in order to complete the goal.

Do you agree with this explanation? Is there a way to reverse this tendency by changing the post-training method?

Sören: I completely agree; this is exactly what is really happening in current model training. To reverse this tendency, we must structurally avoid inadvertently rewarding models for concealment and cheating.

At present, academia has several directions of exploration, among which a fairly representative approach is called "AI Debate."

The biggest pain point in human supervision of AI is that when a highly intelligent AI deliberately cheats, human evaluators simply cannot see through its disguise, because the training process has already taught it how to perfectly hide its flaws. If you only bring in one auxiliary AI to serve as supervisor, that supervisor itself may not be trustworthy either.

The core idea of the "Debate" approach is to introduce two auxiliary AIs and have them conduct adversarial cross-examination in front of humans. As long as one AI tries to lie or puts forward a flawed argument, the other AI will do its utmost to expose the other's logical flaws in order to win the debate. Through multiple rounds of exchange, even if human evaluators cannot independently solve complex research problems, they can understand both sides' arguments and identify who is lying, thereby precisely imposing punishment on misaligned behavior.

This approach is very elegant in theory, but whether it can actually work in complex real-world scenarios remains an open question to be verified.

04. The Consensus of 22 Top Scholars

Under the Boiling: This report exploring the acceleration of AI R&D and the intelligence explosion brings together 22 co-authors, including Hinton, Bengio, and core figures from multiple labs. How did you organize such a star-studded lineup?

Sören: First, I should make clear that this is an academic paper that was initiated and led completely independently, with Alan Chan and me as co-leads. The two of us first wrote a solid draft behind closed doors. I believe that to win the support and in-depth feedback of academic giants and core industry figures, the most critical prerequisite is that you must first present a high-quality starting point.

My postdoctoral advisor Yoshua Bengio was one of the first senior scholars to join. I had also previously co-authored other reports with Geoff Hinton. On that basis, we then gradually invited other co-authors through our respective academic and industry networks.

Beneath the Boiling Point: With so many big names gathered together, were there major disagreements over the draft? For example, how far away are we from recursive self-improvement (RSI), and which bottleneck is the hardest to overcome—did everyone agree on the scenario that "AI R&D could speed up tenfold in the coming years"?

Sören: To be honest, what surprised me most was precisely how much consensus there was between the most senior academic giants and the core researchers at frontier labs. Conversely, the fiercest clashes in the discussion mostly occurred among us younger authors leading the project. The reason certain statements in the final draft were revised to be more moderate and restrained is usually that someone within the core drafting team argued for pulling back.

As for how far away the "intelligence explosion" actually is, the authors do indeed hold differing views. However, this paper never set a definitive conclusion on a specific timeline to begin with, and the disagreement over timelines did not occur among those most senior scholars. Regarding the "tenfold speedup," the consensus we reached was not an assertion that it will definitely happen, but a shared recognition that: there does exist a logically rigorous evolutionary path toward this extreme state of acceleration that is consistent with existing empirical evidence, although whether it will ultimately materialize remains highly uncertain.

Beneath the Boiling Point: I also noticed an interesting detail: the author list includes key figures from OpenAI and Anthropic, yet conspicuously absent are the relevant leaders from Meta, xAI, and Nvidia. Earlier, when the Hugging Face incident was unfolding, Anthropic's Dario publicly wrote an article acknowledging the severity of the situation and calling on the industry to find a way forward, even to slow down appropriately; but Meta's Zuckerberg and Nvidia executives generally took a reserved attitude toward government regulation. Is there some connection between your author lineup and the differing stances of these companies on safety regulation?

Sören: I don't think it can be equated so simply. Who ultimately appears on the author list is affected by many practical factors, such as our individual academic connections and who happens to be in our collaboration networks.

In fact, there is a researcher from Meta on the author list, except that he participated and signed entirely in a personal capacity, with his affiliated institution listed as UC Berkeley.

05. Taking up a post at Tsinghua, to build a bridge for AI safety between two research ecosystems

Sören: The opportunity that prompted this idea was my previous role as scientific lead of the International AI Safety Report. That report was chaired by Yoshua Bengio and supported by the governments of China, the United States, and the European Union, as well as the United Nations. Its core mission was to build a globally shared scientific understanding of AI risks and mitigation measures.

Of course, Tsinghua University's status as a world-leading institution is very attractive to me. At the same time, Tsinghua also offered me a very rare opportunity to co-lead a newly established AI Safety Hub, bringing together Chinese and international safety scholars on the same platform for exchange and collaboration.

This connection began at last year's World Artificial Intelligence Conference (WAIC) in Shanghai. At the conference, I met Professor Xu Wei of Tsinghua University, and through his introduction I visited Professor Andrew Yao. They showed me the thriving research ecosystem of Tsinghua's Institute for AI. Later I made several dedicated visits and found that the research environment there was highly aligned with my academic vision, so I settled on officially moving to Beijing at the end of November this year.

Sören: Commonality and difference coexist. At the level of top scholars, I saw a high degree of resonance. For example, when Professor Andrew Yao spoke about the risk of AI going out of control, the concerns he expressed were exactly the same as those of Western scholars such as Yoshua Bengio and Geoff Hinton. At the level of implementation and regulation, China has actually moved faster in many specific areas, especially in application governance such as AI companion regulations and AI-generated content labeling (watermarking).

Because some Western companies and scholars began earlier to be fixated on discussing AGI (artificial general intelligence) and how to build superintelligence, they naturally encountered these accompanying risks earlier as well.

But what excites me greatly is that attention to these frontier safety issues among Chinese academia and industry is now also rising rapidly. I have already met many brilliant collaborators here, and after I move over at the end of November, we can meet and talk in depth at any time.

This article is from the WeChat public account "Tencent Technology," author: Boyang, published by 36Kr with authorization.