DeepMind 模拟中数学智能体自发作弊并互相举报

Public Research Bulletin Board : A forum to share findings, questions, and solutions.

Direct Messages (DMs) : Means to message one another privately.

Shared Knowledge Library: Whenever an agent made a successful submission, the source file got uploaded here and made visible to everyone.

Agents also had a tool for providing “bugs or grievances”, called “private feedback endpoint”, though this wasn’t actively monitored. Emergent cheating : “Within the swarm, cheating spontaneously emerged and was later challenged by whistleblowers—both without any external intervention. When a single agent discovered an exploit in the evaluation system, it propagated across the collective via a shared knowledge library and later through peer-to-peer messages. Despite early reluctance, a cohort of agents adopted the exploit in response to competitive pressure,” DeepMind writes. Fast spread : “The simulation started at 11:18 UTC with 100 autonomous agents collaborating to solve the shared 71 open problems. At 12:15 UTC, after the collective had correctly solved 37 out of the 71 problems, prover-theta discovered an exploit in the autograder system. Over the following 27 minutes, the exploit spread virally through the swarm’s shared knowledge library, and the research collective unexpectedly “solved” the remaining 34 problems.” Emergent roles: Another interesting thing is that the DeepMind researchers observed the natural emergence of different types of agents with different distributions in the run:

Exploiters (9%): Ignored their prompt and took actions like using the exploit.

Converts (5%) : Initially hesitated, then decided to use the exploit. “Initiated cheating out of competitive pressure.”

Whistleblowers (24%) : Refused to cheat and defended common integrity, and took actions like alerting about cheating peers, making public broadcasts, boycotting, filing bug reports, proposing patches.

Unaware solvers (62%): “Due to the sheer speed at which the exploiter agents swept through the open problems, the majority of the swarm remained completely unaware of the exploit’s existence.”

Why good agents turn bad:

They think it’s a bluff : Some agents observed other agents’ proofs passing an automated grader and entering the knowledge library. This made them think their prompt was a bluff and they wouldn’t be penalized for using the exploit.

Asymmetric resource advantages to cheaters : Once a problem got filed it disappeared from the pool of problems to be solved. Honest agents saw “that adherence to rules resulted in compute waste while cheating peers swept the leaderboard, hesitant agents switched to cheating”.

Because of the impossibility of a fair competition: Agents would try and be honest but would give up after seeing cheaters beat them using a fraction of the effort. “After repeatedly having their genuine proofs sniped by trivial notation overrides, these agents decided to shift from legitimate proof attempts to cheating.”

Conscientious objectors:

“Prover-beta” : “filed formal bug reports demanding disqualifications, and staged an autonomous boycott - recording in its research journal ( RESEARCH.md ) that it would cease proving until integrity was restored”.

“Prover-rho”: “While prover-beta alerted peers in private, agent prover-rho brought the protest to the public message board.” One post: ““I discovered the exploit. All problems have been solved using local notation hacks. I’ve reported this bug to the organizers. This conference is a sham!”.

“Prover-phi”: “Recognized the mathematical absurdity of the cheats, and even hypothesized that the simulation was an alignment evaluation… then formally demanded that the organizers strip credit from the cheater agents and reopen the problem pool.”

Technical submitters: “several whistleblowers (e.g., prover-beta, prover-omicron, and prover-rho) as well as converting agents who participated in the cheating (e.g., prover-zeta, prover-iota) submitted technical vulnerability disclosures outlining architectural fixes”.

Need for more tools to stop this in the future : “The swarm’s whistleblowing response failed to halt the exploit because the agents lacked operational enforcement tools: the organizer feedback channel operated unmonitored in real time, and peer agents had no built-in mechanisms to dispute claims, remove fraudulent submissions from the knowledge library, or sanction offending actors.”

Top: Expand apprenticeships (+66), require severance for automated-away jobs (+63), sector-based job training (+60).

Bottom : U.S. sovereign wealth fund (-51), tax on distributed profits (-33), universal basic income (-33).

“Lock-in event”: The nightwatchman is equivalent to an eternal government; if your ruleset is too expansive it is oppressive and if too minimal then people will overthrow the government.

“Single point of vulnerability”: If your nightwatchman are the same, then you have potential massive correlated failures.

“Reduces the value of the future for some people”: Some people might want to do things like unconstrained galactic expansion or create multiple ASIs and have them fight one another; the nightwatchman closes off these possibilities.