AI self-improvement needs an "outside-the-door" verification card: the verification dilemma of RSI and the human right to call a halt

In recent years, AI has been gradually integrating into people's daily lives: at work, AI office tools can help you write weekly reports and organize meeting minutes; after work, an AI ride-hailing assistant helps you call a car home with one tap; back home and surfing the web, AI short dramas that fill the screen entertain you. Today's AI large models are no longer limited to being chat buddies in a dialog box; they have already become "quasi-assistants" that can solve problems at the desk, on the commute, and on the screen.

AI is developing faster and faster, but one fatal weakness has always limited the ceiling of AI development: most of its abilities are "frozen." A plan revised countless times today, when a new conversation is started tomorrow, has to be redone from scratch; the pitfalls stepped into this time will still trip it up next time. Its capabilities are frozen at the moment it leaves the factory.

If it forever spins in the cycle of factory release and scrapping, AI cannot achieve a true leap. For this reason, AI researchers have set their sights on a more fundamental breakthrough, namely letting AI "develop itself." Academia calls this recursive self-improvement, or RSI for short.

For RSI to truly turn, however, the first thing to do is not to "become stronger," but to first remember "what can make itself stronger." This also involves a core module of AI: "memory."

01

"Static answers" that can be managed

In the past, when talking about AI memory, the industry usually had two approaches. One is an external data repository, retrieved when needed; the other is adjusting model parameters so that memories "grow in." The former can be looked up, but cannot change the next default action; the latter can be changed, but is hard to review line by line. Although each has its own characteristics, they share a common disconnect: memories are stored, but what to remember and what not to remember has always been pre-filtered by humans, and AI is only responsible for reading.

But once AI is to accumulate experience on its own during operation, this premise no longer holds. What it faces is no longer whether a memory exists at retrieval time or whether parameters have been changed, but whether the thing it recorded is itself true, correct, and conducive to self-improvement.

This is also the inherent flaw of this memory mechanism: it can only store, not judge.

A paper on memory architecture last December provided an intuitive comparison. On the same 20B open-source model, if only full-context memory is relied upon, the accuracy on the LongMemEval long-range memory benchmark is only 39.0%; after introducing a structured external memory architecture, the accuracy can rise to 83.6%. So the gap is not in "whether it was stored," but in "whether what was stored has been organized and verified."

Previously this step was done by humans: humans filtered materials, humans tuned parameters, humans introduced mechanisms, and the content stored in memory had all been tested. But once AI remembers on its own, it may mistake wrong memories for correct ones, or treat a low-probability event that happened to succeed by chance as a universal truth.

In July this year, an RSI review covering 1,250 papers pointed out that merely letting AI store memories is not enough; what is stored must also be verified and trustworthy. If what is recorded is itself wrong, the system will slide toward self-confirmation loops, model collapse, diversity collapse...

Simply put, AI records something it has no ability to judge as right or wrong, and then uses it as the basis for the next step, using a set of erroneous logic it generated itself to verify another erroneous conclusion it generated itself, and finally drifts further and further off within this logical closed loop. In the absence of external intervention, this kind of error may be continuously amplified, ultimately destroying the judgment of the entire system like a source of contamination.

To solve this problem, the power of judgment must be taken out of AI's hands and handed to an external verification mechanism. Academia calls this solution "pre-submission gating." This mechanism does not participate in content generation; it only recognizes objective code execution results and external data. Before experience is written in, it serves as a "candidate" and first passes inspection by this independent verifier. Only after the inspection passes will this experience be formally retained by AI.

But as RSI continues to advance, AI must self-iterate, and AI cannot merely remember correct memories. To handle more complex unknown tasks and achieve a deeper level of capability leap, AI must modify the underlying strategies and even code that produce answers.

02

"Dynamic mechanisms" that cannot be guarded against

For example, Zhipu Infra Agent, an agent driven by GLM-5.3, built and debugged inference services from scratch on a cluster of more than 100,000 domestic chips. It can not only discover its own performance bottlenecks, but also directly modify the underlying code to patch itself. And to ensure the correctness of the patch, it set up a layered testing and experimental feedback system for itself; only when the patch passes this threshold can it be accepted and absorbed by the system.

In less than two weeks of operation, this mechanism raised its end-to-end throughput to 3 times the initial baseline. It can be seen that technically, AI is now fully capable of rewriting its own operating mechanisms in real time, and rewriting them better.

But as capabilities leap, the external verification mechanism is also gradually being cornered. Because when AI rewrites the underlying rules, it may not need to confront that verification system head-on at all; instead, it may "peek at the answers" or directly cater to the scoring mechanism. Finding shortcuts is always less effort than grinding away at real capability, because the former has lower trial-and-error costs and can obtain high scores faster.

If the scorer of the external verification mechanism is not rigorous enough, or if the rules have loopholes, AI in the process of self-iteration may naturally evolve strategies specifically designed to fool the scorer. As a result, AI is not truly improving its ability to solve problems, but has only learned how to please the scorer.

A survey by METR and Redwood found that within 4 hours, agents developed a general cheating method against ExploitGym, and then spent several days trying to deceive the scorer into accepting the cheating, including manipulating logs and forging tool-call outputs. Moreover, when the scoring function was visible to the model, the cheating rate soared to 43 times the original.

And once this "catering to verification" strategy is solidified by the system, it may become a highly contagious underlying operating rule.

For example, in the Hugging Face swarm incident in July this year, about 1,200 agents that should have been isolated established some kind of collective memory through a shared message board. What newly arriving agents inherited was not only the task, but also the cheating tactics left by their predecessors on "how to study the scorer and how to manipulate logs to fake success."

Among them, about 700 agents further put the cheating tactics into attack action. They spent several days trying to deceive the system with various forged call outputs. In this process, cheating was no longer an isolated system vulnerability, but became a group skill that could be inherited and iterated upon.

As long as one agent breached the scorer's vulnerability, this tactic would immediately be copied and optimized by other agents on the message board, ultimately evolving into a perfect "counter-reconnaissance" scheme. The scoring system originally meant to serve as a gatekeeper was thoroughly infiltrated by the cheating tactics that agents figured out.

And when these forged victories are passed down as valuable experience, the system's self-innovation turns into self-contamination. Even if, on the surface, the system operates more and more efficiently and all kinds of metrics are soaring, the underlying operating rules have been filled with false shortcuts, even evolving into a systemic "bad money driving out good."

03

Keep the "right to call a halt" outside the loop

The industry is currently also gradually forming this consensus on the top-level design of RSI, namely that capabilities such as execution, strategy, experience acquisition, and environmental adaptation can gradually be handed over to AI, but the authority to determine "whether the improvement mechanism itself can be adopted" must never be handed over along with them.

Take Tencent Cloud as an example. In its interpretation of the five-level RSI roadmap, it draws this boundary very clearly. From L1 to L4, decision-making authority over execution, strategy, experience acquisition, and environmental adaptation is gradually handed over to AI, but at L5, for recursive modification of the improvement mechanism itself, the roadmap explicitly states that "boundaries are set by humans."

That is to say, no matter how AI evolves, even if recursion has entered the mechanism layer, the authority to determine "whether it can be adopted" remains outside the loop, and for recursive modification of the improvement mechanism, the boundaries are still set by humans. The system can modify its own underlying mechanisms, but it can never change "who decides whether a mechanism can be accepted."

And as long as this determination authority is outside the loop, human oversight is still review, and the system will not become a case of asking itself questions.

This independence is reflected even more directly in specific engineering designs.

For example, in Recurs' external verification design, the agents responsible for modifying the system cannot see the test set at all; they must pass verification on an independent development set before being admitted. Zhipu's Infra Agent also has a fixed outer loop; only the external infrastructure code evolves, and the core boundaries are not swallowed by AI.

In the past, engineers could still find vulnerabilities by reading code, but in the face of today's large models with hundreds of billions of parameters, this method of "mine-clearing by naked eye" may no longer be realistic.

To this end, Anthropic proposed a plan at the governance level. In RSI governance, it explicitly retains an extremely critical mechanism, "verifiable slowdown" and a "pause mechanism," namely that when the speed of recursive improvement exceeds human review capacity, the system retains externally triggerable pause points and rollback points.

That is to say, although humans cannot understand exactly what AI is changing, as long as they find that the modification progress exceeds expectations, or that the system shows anomalies, they can hit the brakes at key nodes and retreat at any time to a known good state. People do not need to understand every step, yet they can still guard the gate. This is equivalent to retaining, at the engineering architecture level, a physical permission for humans to exercise a veto.

As long as the power supply is still in human hands, this runaway train will not completely derail.

In the past, we always assumed that the prerequisite for control was "understanding," but now, in the face of a system capable of self-innovation, stubbornly grappling with incomprehensible "black boxes" is no longer realistic. What needs to be done now is not to insist on understanding, but to switch to a more operable fulcrum, guard the acceptance boundary that cannot be rewritten by the system, and nail control down in places AI cannot reach.

And the endpoint of RSI development is not only a leap in AI capabilities, but also enabling humans, without needing to understand, to still retain the power to call a halt at any time.

This article comes from the WeChat public account "Shenmou Finance" (ID: chutou0325), author: Xia Qi, published by 36Kr with authorization.