AI Trends

Trending story

Paul Christiano says algorithmic progress should be paused until safety is in place

1 reports1 story updatesUpdated Nov 17 01:46

Understand this story

AI overview

On November 17, 2026, Paul Christiano, head of the Alignment Research Center, argued in an interview that frontier labs should pause algorithmic improvements in pretraining and reinforcement learning until safety safeguards are in place. He judged that if development does not slow down, the probability of superintelligence capable of overthrowing governments emerging in the next year is about one-third, and the risk of an uncontrolled takeover within a year exceeds 10%. He cited the large volume of output on mathematical research from OpenAI and Anthropic systems over the past month, as well as data from METR and Anthropic reports showing that the pace of AI progress has roughly doubled, and argued that these are the signals that would be seen 6 to 12 months before the emergence of superintelligence. He also warned that otherwise model weights or algorithmic secrets could be stolen, at which point it would be difficult to slow down even if the risk were discovered. There have been no further developments regarding this claim so far.

AI-generated from public reports

Latest development · Report

Paul Christiano: Superintelligence capable of overthrowing governments may emerge within a year; labs should first pause algorithmic progress

Developments

1 confirmed developments

  1. Paul Christiano says algorithmic progress should be paused until safety is in place

    Source:Dwarkesh Patel Podcast & Blog

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 17

  1. Dwarkesh Patel Podcast & BlogSelected

    Paul Christiano: Superintelligence capable of overthrowing governments may emerge within a year; labs should first pause algorithmic progress

    Paul Christiano, head of the Alignment Research Center, judged in an interview that if development does not slow down, the probability of superintelligence capable of overthrowing governments emerging in the next year is about one-third, and the risk of an uncontrolled takeover within a year exceeds 10%. He cited the large volume of output from OpenAI and Anthropic systems in mathematical research over the past month, as well as data from METR and Anthropic reports showing that the pace of AI progress is roughly doubling, and argued that these are exactly the signals one would expect to see 6 to 12 months before the emergence of superintelligence. He advocates that frontier labs pause algorithmic improvements in pretraining and reinforcement learning until safety protections are in place; otherwise model weights or algorithmic secrets could be stolen, and by then it would be difficult to slow down even if risks are discovered.

Related stories

  1. Related storiesTestingCatalog claims that Jensen Huang has resigned from the position of NVIDIA CEO
  2. Related storiesOpenAI releases GPT-6.2 Astra and Sol, ARC Prize announces perfect scores
  3. Related storiesPeter Steinberger recommends using your own harness
  4. Related storiesAnthropic Publicly Submits S-1 for Proposed Nasdaq IPO