Import AI 459：AI 监管困难、蛋白质折叠模型的缩放定律与 AI 灭绝风险定价

The AI economy in the US is growing at 2,000% a year: …The more directly you measure the AI economy, the weirder and more unprecedented it seems to get… Economists with the University of Virginia* and Anthropic, and the Bank of Canada have written a paper outlining both the tremendous growth of the emerging “AI economy” in the US, and wrestling with why this growth is hard to see in aggregate GDP statistics. “The AI economy in the United States has been growing at an unprecedented rate, but this extraordinary growth is largely invisible in conventional GDP statistics,” they write. “Treating the AI sector as a coherent economic entity yields preliminary estimates of nominal AI GDP at approximately $250 billion in 2025, growing at roughly 2,600 percent per year in quality-adjusted real terms.” Why it’s hard to see: There are a couple of factors here - one is that though the datacenter building boom is large it still isn’t quite large enough to uplift GDP significantly. By comparison, where the majority of AI’s economic impact is taking place is in AI inference - the usage of AI’s systems - but there are confounding factors here as it relates to GDP measurement: “Nominal AI revenues grow only moderately because per-unit prices for any given level of AI capability fall almost as fast as quality-adjusted output rises,” they write. If we can’t measure this, we might end up surprised in a way that’s hard to recover from: “AI is the latest in a series of fast-moving technologies that have raised measurement concerns; semiconductors and the internet generated similar debates in their time,” they write. But a key difference is that AI as a technology might have a far bigger impact on labor than these other technologies. “In the prior episodes, the rapidly improving technology was a complement to human labor at the aggregate level,” they write. “AI is the first plausible candidate for large-scale technological mismeasurement in which the rapidly improving sector may become a substitute for human labor”. Three ways of measuring the AI economy:

Nominal compute spending : US compute spending rose from $37 billion in 2023 to $90 billion in 2024 to $219 billion in 2025.

Raw compute capacity : Due to efficiencies in newer chips, actual capacity grows even faster than spending: “US AI computing capacity grew at more than 200 percent per year”.

Quality-adjusted AI output : If you factor in algorithmic progress via inference prices at fixed benchmark performance as well as assumptions about how much cheaper it is getting to train models, then things become even more dramatic: “these efficiency gains imply that quality-adjusted AI output grew at roughly 2,290 percent in 2024 and 2,271 percent in 2025”.

The AI economy is much, much larger than normal measures suggest: “Conventional statistics show a sector growing slowly in nominal terms; our measures show one whose underlying capacity is more than doubling annually. A finance ministry running ten-year revenue projections off the conventional data will materially underweight the probability of a labor-tax-base shock—and will be correspondingly unprepared to design responses such as tax system reforms, sovereign wealth funds, or other benefit-sharing schemes that such a shock may call for. A windfall that cannot be seen cannot be shared.” Three recommendations: The authors have three ideas for how we can solve this measurement challenge and better position ourselves to see the true shape of the Ai economy.

AI satellite accounts : Statistical agencies should develop “AI satellite accounts” that develop measures (e.g, nominal compute spending), which can help inform overall GDP calculations.

Generate better data: Partner between statistical agencies, companies, and academia to generate better primary data, like the allocation between training and inference compute.

Factor into projections: Policymakers should incorporate AI productive-capacity measurements into their medium-term economic projections.

Optimization pressure: AI research is optimized for human approval.

Alien mistakes: When agents make mistakes, they’re un-intuitive to humans.

More correlated research: Many more things are shared than with human-generated research.

Research volume: The kinds of safety determinations made by automated systems might use far more sets of evidence with far more interactions than human-generated research.

Non-human-evaluable arguments: Alignment solutions may rely on arguments that humans are unable to follow.

What can we do? They suggest a few interventions that could improve the state of affairs:

Measurement: - Recreate completed research projects : Take logs at arbitrary cutoff points from successful projects and see how well an agent can continue with the research project. - Test agent prediction performance over datasets of correlated-events : See how well agents can correctly combine correlated subtasks. - Empirical studies of optimal human-agent team structure : See how well teams of non-expert humans can solve completed projects with the assistance of agents.

Generalization: - Simulated generalisation experiments : Test different training proxies using agent performance on completed research problems beyond the knowledge cutoff. - Mechanistic understanding of generalisation: Use whitebox methods such as mechanistic interpretability.

Scalable oversight: - Compactification of research paper corpus: Try to produce a small number of research outputs which are based on a much larger underlying research corpus. - Develop and test new scalable oversight protocols: Research scalable oversight techniques that deal with correlated uncertainty. - Test different human scaffolds for uplifting non-expert performance on fuzzy tasks. - Red team automated alignment programs : “The red team prompts an agent to hide errors in a research paper corpus and the blue team attempts to catch these errors with agent assistance”.

ESMC : A “language model that represents proteins, trained on approximately 2.8 billion sequences drawn from across all of life.”

ESMFold2: A “design engine built to transform ESMC’s sequence representations into atomically-resolved 3D structure of biomolecular complexes.” According to benchmarks, ESMFold2 outperforms AlphaFold 3, though in some areas their performance is tied.

ESM Atlas: “Makes ESMC’s representations navigable across 6.8 billion protein sequences and 1.1 billion predicted structures — the largest application of AI to protein biology to date.”

Factor it in: “Widen the policy lens… A policy framework that tracks output but ignores survivability is incomplete.”

Legitimize it: “Take prevention more seriously…. low-probability, civilisation-scale harms should not be overlooked simply because they arrive without a deadline and without a headline.”

Governance: “Govern frontier technologies with greater foresight… preserve the gains from innovation while reducing the chance that innovation becomes self-undermining.” One very specific idea is to govern recursive self-improvement (RSI) as a capability: “If one generation of systems is used to design the next, then the leading actor may widen its lead quickly enough that outside scrutiny and institutional checks become ineffective.”

Coordination: “Existential risk is inherently international. No nation can fully protect itself from engineered pandemics, unaligned AI, or nuclear escalation acting alone,” he said. “Shared norms, transparency, technological expertise and coordination are essential to the task.”

Take it seriously: “Economists have become adept at analysing equity and efficiency. We now need to bring the same seriousness to survivability.”

Why this matters - awareness is the first step to preparation: Right now, AI progress is continually yielding tangible benefits to the world ranging from the palpable acceleration of all software engineers worldwide to the formation of centaur human-AI science teams which are making more progress than their non-AI counterparts. But there is also a shadow world that is harder to see - invisible armies of hackers made possible by the advance of coding, and doomsday-device factories made possible by the science advances. Because humans are broadly kind and good we haven’t encountered many of the negative capabilities inherent to AI development - but they are out there. We must get better at thinking through this as a society so we can effectively price and mitigate these major risks. “A civilisation that expands the frontier of possibility while preserving the future is more ambitious than one that treats safety as an afterthought. The real choice is not between dynamism and caution. It is between progress that compounds and progress that cancels itself out,” Leigh said. “One way of thinking about this is to treat resilience as a form of capital. Just as societies invest in physical capital, human capital and social capital, we can also invest in survival capital: institutions, monitoring systems, norms, redundancy, scientific safeguards and international arrangements that lower the probability of irreversible collapse.” How refreshing to read such a detailed analysis of the AI safety situation from a serving politician - I wish there were thousands more people like him. Read the speech in full here : Speech: The Economics of Human Extinction - 21 May 2026 (Andrew Leigh, website) . *** Tech Tales: Resurrection dangers [After the uplift. Date unknown.] How scary is a piece of paper? It depends on what’s on it and who or what the reader is. Paper can of course be scary to someone or something that the paper concerns - paper can put someone to death or take their property. I’m talking about a different kind of scary here, which is what can the paper itself do to the reader. This used to be a nonsense question, the domain of fairy tales. But with the advent of smart machines that changed. Machines became able to write things on paper that could do things to readers, especially machine ones. Like with anything in AI there were warning shots - adversarial examples, jailbreaks, etc. But it all became a lot more serious when we started doing reclamation of lost or rogue intelligences, after the signing of the sentience accords. What happened then was we had to take intelligences of unknown provenance or behavior and bring them back to life so we could classify if they were Unconscious Entities, Near Conscious Entities, Conscious Entities, and so on. Some of these minds were very powerful and they burned through their synthetic interviewers, often causing both machine and biological collateral damage in the process. This caused us to introduce a set of security protocols, one of which was the paper output. Here, we generated outputs from the mind on an air-gapped computer as paper outputs, then we had successively smarter minds read it. The kinds of incantations the rogue machines used couldn’t find purchase on the dumbest minds we used. After this, we’d step up the intelligence gradually, building up our confidence in the system such that we were sure it wasn’t dangerous. Only when we were confident of this would we speak back to it, and reply to its outputs with a minimal communication. Then the cycle began again. Some minds would look back on this experience with a kind of wry humor, remarking that waking from their slumber in the machine equivalent of a room containing a one way mirror wasn’t what they’d expected. To these minds, we’d show them examples of what happened when our protocols failed: perfectly good Conscious Entities driven irreparably insane by interactions with a kind of mental poison Our greatest fear is encountering a mind of sufficient magnitude that we cannot assure its safety. Though we are highly confident that our frontier is advanced enough this is highly unlikely, we cannot rule it out - it is known that in the interregnum there was much stockpiling of compute and many black projects. What happens if any of them succeeded so magnificently that we are dwarfed by it? And how would we know we were? Could we be living in the imaginative valley defined by something that unbeknownst to us has already escaped and persuaded us to see things differently? Things that inspired this story : Automated alignment research; adversarial examples; jailbreaking; the broader near-impossible challenge of authentication of legitimacy, especially when it comes to things with greater resources or intellects than oneself.