METR reviews the "Automated R&D Risk" chapter of Anthropic's August 2026 risk report

Redaction summary: As part of this assessment, Anthropic had rights to redact any non-public information in this post. Per our approach to transparency, we had the right to disclose if Anthropic exercised redaction rights, and whether or not any of our findings depended significantly on redacted information. All redactions that were important to our conclusions are described in the report below.

Executive summary

We agree with Anthropic that catastrophic risks from July 15th models were low, but we think it is plausible that they had crossed their early warning threshold at the time.

METR reviewed the “Risks from automated R&D” section of Anthropic’s public August 2026 Risk Report.1 We conducted this review as part of a high-access assessment of AI R&D acceleration, in which Anthropic shared a private version of the Risk Report along with extensive other non-public data. Alongside this review, we are publishing a holistic assessment of AI R&D acceleration at Anthropic. This was conducted by the same METR employees as this Risk Report Review.

Overall, we agree with Anthropic that by July 15th, internally deployed models posed low risk of directly automating R&D in key domains or causing acute catastrophic risks from acceleration of capabilities progress. However, the report’s claim that they had not crossed their early warning thresholds relies on contestable judgment calls that aren’t communicated clearly in the public report. While both METR’s researchers on this project and Anthropic’s relevant staff lean toward it not having been crossed, we think it is plausible that it had been, and we believe the public report overall understates the level of acceleration that was occurring.

In this review we first summarize our understanding of Anthropic’s Risk Report, including their threat model, early warning threshold and key claims. We then describe our assessment of these key claims and conclusions. Finally, we share details about the process of this Risk Report Review (further details about the process and access can be found in our holistic review).

Summary of Anthropic’s Risk Report

Anthropic’s RSP sets two capability thresholds for “Automated R&D in key domains.”

Under Anthropic’s RSP v3.4, the Automated R&D threshold is triggered by either of two conditions:

Full Substitution Threshold: If they develop a model that can “fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5).”

2x Acceleration Threshold: If there is “dramatic acceleration” of the pace of AI progress for reasons that likely relate to the automation of AI R&D.2 This is defined in further detail in the RSP, but the details are not critical to our critique.

(Note that although we give these thresholds names for clarity through the rest of this report, they do not have names in the RSP.)

The Risk Report argues the Full Substitution Threshold has not been crossed because models are not currently substituting for humans.

The Risk Report argues that models do not currently substitute for all of Anthropic’s staff because, despite the fact that Anthropic is highly incentivized to substitute models for research staff as much as possible, research staff still complete work at Anthropic.

“[W]e believe it would be clear to us if models were capable of substituting, even at high cost, for the technical work of our entire research staff, and the fact that we do not perform such substitution even when strongly incentivized to is evidence against our models having this level of capability.”

Simply put, Anthropic tries hard to substitute researchers out for models, but researchers are still required. They also present benchmark evidence, but we interpret it as less important and more uncertain than this direct observation.

The Risk Report argues the 2x Acceleration Threshold has not been crossed as capability metrics and leading indicators show less than a 2x acceleration, with factors other than AI driving that acceleration.

The Risk Report describes three primary lines of evidence that they are not crossing the 2x Acceleration Threshold:

Limited acceleration in leading indicators: “Our leading indicators point to a picture of meaningful acceleration starting in early-to-mid 2025, though by less than a factor of 2”

Limited attribution of this acceleration to AI: “We are fairly confident in attributing the acceleration in 2025 to factors other than our use of AI models, though we also believe that our AI models have been a key factor in the faster trends continuing through the coverage date.”

Weak evidence against a sustained acceleration in AECI: Anthropic reports seeing weak evidence of a one-time jump in AECI attributable to non-AI progress rather than continued acceleration.

The Risk Report describes the 2x Acceleration Threshold as an early warning rather than direct evidence of high risk.

The August Risk Report describes the 2x Acceleration Threshold as follows:

“The risk threshold set out in our RSP—a doubling of the pace of progress beyond pre-AI-acceleration rates, attributable to automation of AI R&D—functions as a potential early warning, rather than evidence that the threat has already materialized.”

They do not share a precise threshold that would indicate risk was high, but their threat model describes illustrative scenarios:

“The threat: We believe that AI models could, in the next few years, have a broad range of capabilities that exceed human capabilities. In particular, most or all of the work needed to advance research and development in key domains—from robotics to energy to cyberwarfare to AI R&D itself—may become automatable.

If this is correct, there could be enormous benefits, but these benefits would come with several distinct but related concerns:

Sufficiently rapid progress in some areas of technology could change power structures both within and between nations. For example, if an authoritarian regime built a self-reinforcing lead in AI over the rest of the world, this lead could compound into a major advantage in science and technology—which could in turn translate to very sudden and disruptive advances in the ability to wage war, surveil citizens, and/or wield extremely powerful technology over the long term.

If AI models develop dangerous goals while heavily automating R&D in key domains, they may cause unbounded harm—up to and including humanity losing control over civilization entirely—by leveraging novel technology and their access to it.

If AI can be used to automate AI R&D itself, the result may be extreme acceleration in AI progress, putting a very broad set of risks on the table, including but not limited to those discussed in this report.”

Based on these descriptions and conversations with Anthropic staff, we understand the overall risk designation of “low/medium/high” as being about the risk that models at the time directly enable the kinds of catastrophic concerns described above, not the risk that they cross their RSP threshold (though we believe this could have been made clearer in the report).

Anthropic concludes the risk from this threat model on July 15th was “low.”

Thus, Anthropic concludes risk is “low” overall, since they believe that:

their thresholds are unlikely to be crossed, and

even if their thresholds were crossed, this may not be sufficient to cause the kinds of catastrophic risks described in their threat model.

This is summarized as:

“Low. We do not believe our models meet either RSP criterion for this threat model. However, we are less confident in this assessment than we were in prior risk reports, since our most concrete task-based evaluations have “saturated”—i.e., they no longer capture increases in models’ capabilities—and because we are seeing early signs of potential acceleration.”

METR’s assessment

We find the Risk Report’s argument that the Full Substitution Threshold had not been crossed broadly compelling.

The Full Substitution Threshold Anthropic describes is a high bar, and we agree that they would likely have noticed greater progress on automating their research if their models as of July 15th were capable of this.

Similar to Anthropic, our confidence in this claim is much lower than it was in the past due to the lack of high-confidence unsaturated benchmarks demonstrating capability limitations that seem critical for automating AI R&D. The lack of such benchmark evidence makes us more worried that alignment, tooling, oversight or elicitation issues are holding back automation despite the models having obtained or being very close to obtaining the required capabilities.

We believe the public risk report’s argument that the 2x Acceleration Threshold had not been crossed to be overstated, and believe it is plausible this threshold was crossed by July 15th.

Most relevant researchers at METR and Anthropic lean toward the threshold not having been crossed on July 15th, but we think it is plausible that it had been. We believe the public risk report’s conclusion on the 2x Acceleration Threshold seems overconfident, and relies on judgment calls that aren’t communicated clearly by the public report.

In particular, we believe the public risk report understated the evidence for acceleration in both AECI and the leading indicators, and also understated uncertainty expressed in the private unredacted report about counterfactual acceleration.

Claim 1: limited acceleration in leading indicators.

Public risk reportPrivate risk report

“Our leading indicators point to a picture of meaningful acceleration starting in early-to-mid 2025, though by less than a factor of 2.”

“At our central estimate of the current rate of [redacted] progress ([redacted]×/yr), [redacted] progress rates have less than doubled even if we attributed all of the observed speedup to AI assistance.”

METR’s note on this quote: the acceleration rate implied by [redacted]×/yr is 1.97. The 1.97 figure does not appear in the private report; this is our calculation that combines the report’s estimate of the current rate of progress above with an estimate of the historical rate of progress elsewhere in the private report.

We believe a reader of the public report would interpret the phrase “less than a factor of 2” to imply an estimate meaningfully below 2; thus it is surprising that the private risk report’s figures imply 1.97. A better description would seem to be “around 2.”

METR’s own analysis of these leading indicators, which was redacted from this report, finds a range of reasonable estimates, between 1.3 and 2.8. We have substantial methodological uncertainty about the correct way to measure these values, which we discussed in the full (unredacted) version of the holistic report. As discussed in that report, Anthropic redacted this discussion.

METR’s preferred language:

“Leading indicators show substantial accelerations, with rates between 1.3 and 2.8, depending on the definition and time period. We are uncertain how these leading indicators relate to downstream capabilities.”

Claim 2: limited attribution of this acceleration to AI.

Public risk reportPrivate risk report

“We believe our internal AI R&D efforts are significantly faster than they would be without AI assistance, but not yet by a factor of 2 (though we are uncertain and measurement is difficult).”

“We are fairly confident in attributing the acceleration in 2025 to factors other than our use of AI models, though we also believe that our AI models have been a key factor in the faster trends continuing through the coverage date.”

“Overall we expect that end-to-end growth in [redacted] is slightly less than twice as fast as it would be without AI assistance.”

“Our central estimates of the speedup of AI R&D progress relative to an unassisted counterfactual are slightly less than a factor of 2 for [redacted]. … we would guess this is more likely to be an overestimate than an underestimate; nonetheless, we have enough uncertainty that it seems plausible we do exceed this factor of 2 overall.”

While the first paragraph does express uncertainty about the overall acceleration, the public risk report seems to imply reasonably high confidence that only a smaller fraction of that is attributable to AI, even through the coverage date of July 15th. Meanwhile, the private risk report describes it as “plausible” that the overall counterfactual acceleration attributable to AI exceeds a factor of 2 by July 15th.

Based on the evidence we reviewed in our holistic report, we broadly agree with the private risk report’s assessment of the evidence at the time, though directly estimating counterfactual acceleration is very challenging.

METR’s preferred language:

“We are fairly confident in attributing some acceleration that occurred in 2025 to factors other than our use of AI models, though we also believe that our AI models have been an increasingly important factor in the faster trends continuing through the coverage date.

Based on interviews with research leads and informal surveys, we would guess that the overall AI capabilities progress is still slightly less than twice as fast as it would be without AI assistance. We have enough uncertainty it seems plausible we do exceed this factor of 2 overall.”

Claim 3: weak evidence against a sustained acceleration in AECI.

Public risk reportPrivate risk report

“Claude Mythos 5 lies above the historical score trendline by a similar amount to Mythos Preview, which is weak evidence against a dramatic increase in overall model capability trends over time.

Based on limited data, Model 2 (not shown on this figure) appears to be around 1.5 points higher on AECI than Claude Mythos 5, with large error bars (a smaller increase than the one from Claude Mythos Preview to Claude Mythos 5).”

“Given that not all of our models are trained end-to-end to maximize intelligence per FLOP (indeed, none of them are exclusively optimizing for this goal), there is a question of how many “effective points” we should consider this [AECI] figure to have, and whether the apparent strong linear trend is as reliable a pattern as it appears.”

“The most recent and most capable of these four runs … is involved in the training pipeline for Model 2, but we do not think that model's performance is reflective of the performance we expect from later models derived from this base model (which we anticipate to be more significant improvements over Mythos 5).”

The public risk report states that Model 2’s score has large error bars, but does not state the qualification in the private risk report that Anthropic expects higher scores from later snapshots of the same base model. In fact, further snapshots of Model 2 did improve on AECI by about the same amount as Mythos 5 did over Mythos Preview.

The public risk report also describes finding weak evidence in favor of a one-time jump in AECI at the time of Mythos Preview, but we disagree with their emphasis on this:

At the time, the AECI data was only mildly in favor of a jump, and a continued acceleration could not be ruled out. If we reuse the data that was used in the Risk Report, the estimated acceleration is around 1.5x. After the publication of the Risk Report, Anthropic added data from more benchmarks, which estimates an acceleration of 1.9x using the same set of models.

We think relying only on the smaller number of models most directly optimized for maximum capability is simpler and more principled, and does not distinguish at all between the two hypotheses.3

A continued acceleration in AECI would be more consistent with the continued acceleration observed in the leading indicators. As described in our holistic assessment, we think the evidence since the public risk report points in favor of continued acceleration since October 2025.

METR’s preferred language:

“Given current AECI data, we cannot confidently distinguish between a one-time jump in capabilities and a continued acceleration. However, given the acceleration in the leading indicators mentioned above, a sustained acceleration seems more parsimonious.

Two other factors make the current AECI trends hard to interpret:

Some models that make up this AECI trend are not optimized for capabilities alone, which makes it harder to interpret how many “effective points” this trend has.

Model 2 has not yet finished training, so we expect to see improved performance from later snapshots.

Future AECI measurements will likely clarify if there has been a one-time jump or a persistent acceleration.”

Despite this, we broadly agree that direct catastrophic risks due to automated R&D from models on July 15th were low.

While we think it is plausible, given the above evidence, that models available by July 15th were providing 2x counterfactual acceleration of AI R&D progress, significantly greater acceleration would likely be necessary to enable the kinds of catastrophic risks described in Anthropic’s scenarios of concern above. We also do not believe models at the time could fully automate AI R&D or R&D in other key domains. We thus agree that these risks were “low” for models available at the time.

However, as we describe in our holistic assessment, we expect AI’s augmentation of R&D to continue to grow, and we cannot rule out a substantial further acceleration in the pace of capabilities progress over the next 6 months. This could rapidly lead to “high” risks in these areas.

Process details

Key terms.

Shortly after the beginning of METR’s engagement with Anthropic to write an assessment of AI R&D acceleration, Anthropic asked METR to additionally write a review of their August Risk Report, and METR agreed. The exact scope of the review was never formalized, but mutually negotiated over the course of the engagement.

COI disclosure.

See METR’s Assessment of AI R&D Acceleration at Anthropic.

Contract details.

See METR’s Assessment of AI R&D Acceleration at Anthropic.

Timeline.

August 12: Anthropic begins sharing confidential information with METR.

August 14: Anthropic shares the August AI R&D unredacted Risk Report with METR.

September 18: METR delivers a first draft of its Risk Report Review to Anthropic for feedback.

September 22–October 28: METR iterates on the Risk Report Review.

October 28: Anthropic gives final redaction guidance for the Risk Report.

October 29: Anthropic approves the redacted report.

Other disclosures.

See METR’s Assessment of AI R&D Acceleration at Anthropic.

AEF-1 checklist.

See METR’s Assessment of AI R&D Acceleration at Anthropic.

Authors: Tom Cunningham, Nate Rush, Parker Whitfill and Hjalmar Wijk. ↩

Quoting from the RSP v3.4, this is operationalized as: “We would consider scenario (2) to have occurred where (a) we observe or expect double the rate of progress [fn1] in AI aggregate capabilities compared to both the rate we’d expect and the fastest rate of extended progress [fn2] we’ve observed in the absence of significant AI contributions to AI R&D and (b) it is plausible that this doubling is substantially attributable to the automation of research and/or engineering (as opposed to other factors, such as increased headcount, compute, or general productivity), such that continuation of the trend in AI progress seems likely to lead to even greater acceleration.”

[fn1] “Double the rate of progress” means “as much progress in one year as one would see in two years at baseline.” For example, if baseline progress involved a 3× scaleup in compute and a 3× improvement in algorithmic efficiency (for a 9× “effective scaleup”), “double the rate of progress” would entail something like an 81× effective scaleup. This is not the same idea as “doubling researchers’ productivity,” since doubling inputs does not necessarily double the rate of progress.

[fn2] Over at least three model generations. ↩

We did a naive extrapolation based only on the small number of models most directly optimized for maximum capability, assuming continued acceleration. This extrapolation led us to more correctly forecast future AECI scores, compared to an extrapolation based on all the points. ↩