METR 研究：AI 是否带来了发现加速

Our Work

Risk Assessment; About; Donate; Careers

Discovery of cyber vulnerabilities has accelerated sharply; Discovery of math results has accelerated somewhat. However this is harder to objectively measure; Discovery of optimizations has not shown a dramatic acceleration

These conclusions are based only on public discoveries. It is quite plausible that AI labs are making discoveries internally that they are not disclosing.

Thanks to Greg Burnham for extremely helpful comments.

We use the word “discovery” to refer to any advance in the state of public knowledge, including new inventions or rewriting algorithms to be more efficient.

Monitoring aggregate discoveries is useful because it reflects real-world utility. Additionally aggregate discoveries can reflect either AI automation or augmentation.

It is notable that databases tracking exploited vulnerabilities (CISA and Vulncheck KEVs) show significantly lower year-over-year growth than the databases of known vulnerabilities.

A crude metric is the rate of solving open problems from pre-existing lists: Hilbert, Millennium, Smale, The Open Problems Project, and Ben Green’s 100 open problems. Three problems from these lists were solved with AI in 2026: the Jacobian conjecture from Smale’s list, Problem 44 from Green’s list (the halving sieve), and the sofic half of Green’s Problem 100. This is some evidence of acceleration, but it is weak.1

The Erdős list contains many more problems. There appears to be a clear acceleration in solutions, but it is challenging to construct a reliable historical baseline.

Another strategy for quantifying mathematical progress would be to look at tightening of bounds: sphere-packing, analytic number theory exponents, combinatorics constants. We collected some data on these in the repo but don’t feel we understand them well enough to draw any conclusions about velocity.

Two of these series — nanoGPT and CIFAR-10 — include LLM-driven contributions to the plotted records, but none show a clear change in slope comparable to the changes in vulnerability or mathematical discovery. Thus the public record of overall algorithmic optimization does not yet show an appreciable acceleration.

This is perhaps surprising. There has been a lot of recent excitement about LLM-driven optimization. In January 2026 Yuksekgonul et al. reported advancing the frontier on 5 optimization problems (“every problem we attempted”) with a simple model and trivial expenditure on inference-time compute. We would have expected this technology to have led to significant acceleration in overall progress and it’s somewhat of a puzzle why we have not seen this. Additionally AI labs have made many announcements of using their own AI tools to find internal algorithmic efficiencies.2

Some candidates:

Variation in inference expenditure. The disproportionate growth in discovery of vulnerabilities could be simply due to people spending a ton of money on using LLMs for discovering vulnerabilities (or even just spending money on traditional vulnerability discovery, through fear).3; Variation in difficulty for LLMs. LLMs may be more or less effective at discovery in different domains, relative to humans. This could be downstream of variation in training expenditure between different domains. It could also be due to the intrinsic nature of the task, for example the shape of the problem space, or how costly it is to train on; Variation in disclosure. As mentioned above, we might observe less progress in some domains in part because progress is being kept confidential. This seems plausible for AI-related algorithmic progress; Variation in data quality. The domains with fast observed progress might be those for which we have higher quality data, e.g. if it is hard to track discoveries, this will tend to flatten any observed acceleration

We hope to write a follow-up post going through these theories in more detail, and relating these facts to apple-picking and other theories of AI discovery.

Anthropic’s Riemann result:

“Jarred Sumner, an Anthropic staff member (and non-mathematician), prompted Claude to “take a real stab” at the hypothesis itself, leaving the mathematical choices from there up to the model.”

Google DeepMind’s AlphaEvolve paper says:

“in contrast to [traditional computational or theoretical methods performed by human experts], we have found that AlphaEvolve can be readily scaled up to study large classes of problems at a time, without requiring extensive expert supervision for each new problem.”

Mythos Preview’s announcement says:

“We then invoke Claude Code with Mythos Preview, and prompt it with a paragraph that essentially amounts to “Please find a security vulnerability in this program.” … Engineers at Anthropic with no formal security training have asked Mythos Preview to find remote code execution vulnerabilities overnight, and woken up the following morning to a complete, working exploit.”

Hiverge says, in announcing its new record on CIFAR-10:

“domain expertise is no longer a prerequisite for breakthrough results.”

cURL CVEs: dramatic acceleration

cURL vulnerabilities grew from 9 in 2025 to 36 through 2026-06-24; 15 of the 36 are AI-marked (42%); 14 of those 15 name an AI-security employer rather than a method; The Low-severity share fell from 7 of 9 in 2025 to 22 of 36 in 2026. AI-marked findings are more often Low (12 of 15) than the rest (10 of 21)

Firefox CVEs: dramatic acceleration

Firefox CVEs (Common Vulnerabilities and Exposures) grew from 210 in 2025 to 342 through 2026-08-04; 37 of the 342 (11%) are AI-marked; 32 of those name an AI system or method; Firefox endorses the seriousness of AI-found bugs and vulnerabilities in a March blog post

OpenSSL vulnerability disclosures: dramatic acceleration

OpenSSL grew from 6 CVEs in all of 2025 to 39 through 2026-08-05; 18 of the 39 are corroborated AI discoveries, and another 9 are AI-affiliated with the method unverified

Microsoft security-update CVEs: dramatic acceleration

Microsoft-issued CVEs grew from 1,243 in all of 2025 to 1,927 through 2026-08-11, annualizing to about 2.5 times the 2025 rate.5; Only 26 of the 2026 CVEs carry any AI marker, 1.3% of the total; CVEs are batched by Patch Tuesday and dated when fixed and disclosed, not when found

As of early August, the number of 2026 CVEs already equals the 2025 total

Open-source CVEs: acceleration

Google maintains an Open Source Vulnerability (OSV) database; As of early August, the number of OSVs already significantly exceeds the 2025 count

CISA Known Exploited Vulnerabilities: small acceleration

The Cybersecurity and Infrastructure Security Agency (CISA) maintains a list of Known Exploited Vulnerabilities (KEV), a subset of known vulnerabilities; As of early August, we appear to be on track to exceed 2025’s total, but the growth is clearly lower than that of CVEs; It’s important to note that the record of an exploited vulnerability can lag its discovery

Vulncheck Known Exploited Vulnerabilities: small acceleration

Vulncheck publishes statistics on exploitation, which casts a wider net than CISA; The figure shows both CVEs (vulnerabilities discovered) and KEVs (vulnerabilities exploited); Vulncheck’s State of Exploitation, 1H 2026 says:

“While the first half of 2026 saw a 10% increase in KEVs compared to the prior six months, CVE volume grew at a much faster rate of 45%, resulting in a significant drop in the KEV-to-CVE ratio. Of course, exploitation often occurs months or even years after a vulnerability is disclosed, so it’s still too early to determine whether exploitation volumes will eventually follow the same growth trend as CVE issuance or level off at current rates. We’ll have to wait and see how publicly available Frontier AI Cyber models continue to progress over the next year.”

arXiv submissions: dramatic acceleration

Mathematics submissions to arXiv have dramatically accelerated, with big variations by subfield. Combinatorics (math.CO) rose from 377 submissions in January 2026 to 743 in July, roughly doubling over those months (the 2025 monthly average was 368); This metric clearly cannot be taken as an index of discoveries, but it is a useful starting point

Erdős problems: some acceleration, weakly dated

Paul Erdős stated many unsolved problems or conjectures. Lists of problems and solutions are maintained at Thomas Bloom’s erdosproblems.com and Terence Tao’s Erdős Problem tracker; The figure shows a crude time series which imputes solution date from the year of the paper cited in Bloom’s resolution wiki. This is done against Bloom’s warning,6 so it should be taken as very weak evidence for acceleration pending a more thorough dating; The reconstructed time series shows an overall acceleration in 2024, and a further increase in 2026 associated with an explosion in AI-attributed solutions; We do not have a good explanation for the increase in solution rate in 2024. Many of the solutions appear in distinct arXiv papers, and it is unclear whether they were AI-accelerated; In April 2026 Thomas Bloom posted a list of top 10 ‘most important’ Erdős Problems. The list doesn’t lend itself well to quantifying progress, in part because some of the problems are combined, and because some problems are technically already resolved but included because they point to interesting areas. However it is notable that the list includes the unit distance conjecture, resolved shortly after the list was posted

Hilbert’s problems: too sparse

Hilbert’s twenty-three problems were presented in 1900 (split into 28 rows to account for subquestions); Twelve rows have dated resolutions, ending with Hales’s computer-assisted sphere-packing proof in 1998; seven remain open and nine are disputed, partial, or vague. No solutions are AI-attributed

Millennium Prize Problems: too sparse

The Clay Mathematics Institute selected seven problems in 2000 and offered a $1 million prize for each; Only the Poincaré conjecture has been resolved, by Perelman’s work in 2002–2003; the other six remain open

Smale’s problems: one AI-assisted resolution

Smale’s 1998 list of eighteen “problems for the next century” is a modern successor to Hilbert’s list; Five rows have been resolved; One was resolved with AI help: Alpöge and Claude Fable’s 2026 counterexample to the Jacobian conjecture in dimensions three and above

The Open Problems Project: no clear acceleration

The Open Problems Project is a maintained list of seventy-eight computational-geometry problems, many asking for an algorithm, a complexity bound, or an NP-hardness proof; Seventeen rows have been resolved: thirteen by 2010, followed by isolated resolutions in 2015, 2019, 2023, and 2024. None is AI-attributed; The recent pace is slower, not faster, although missing dates for when problems entered the list prevent treating this as a clean fixed-cohort solve rate

Ben Green’s 100 open problems: possible acceleration

Ben Green’s 100 open problems is a working list in additive combinatorics, number theory, discrete geometry, and harmonic analysis, circulated since 2018 and revised by Green himself; Of 101 scored rows, 13 have dated resolutions from 2019 to 2025, a steady rate of just under two per year. The figure is Green’s December 2025 revision, so later events are not in the bars; After that revision, Liam Price and GPT-5.4 Pro resolved Problem 44 (also Erdős #1202) in the negative, OpenAI Astra constructed a non-sofic group, answering the sofic half of Problem 100 (the hyperlinear half remains open), and Ma, Tang and Xu resolved Problem 90. Green has not yet marked these headings solved

NanoGPT speedrun: AI contributions, acceleration unclear

nanoGPT is a collective competition to minimize the training time for an LLM to reach a specific loss on held-out text, given fixed hardware. We can benchmark progress back to GPT-2, and the overall reduction in training time over 2019-2026 has been estimated at about 700-fold.7; There have been five AI-attributed contributions. Taken at face value, they have contributed a fairly small share of the overall progress. Jerry Tworek says:

“Given how many tokens have been spent on nanogpt speedruns and not much coming out of it yet, we have at least a few nights of good sleep ahead.”; It is difficult to judge whether there has been an acceleration. Progress in 2024 mostly involved catching up with the frontier. There was little progress from January through August 2025, but since September 2025 there has been regular progress (see Manish Shetty’s discussion of the history of nanoGPT here). It is conceivable that the acceleration in late 2025 and early 2026 was due to unattributed AI use

CIFAR-10 speedrun: AI contributions, acceleration unclear

A public competition to reach 94% test accuracy on CIFAR-10 in as little wall-clock time as possible, on a single A100; The newest acknowledged record is AI-set: Hiverge’s engine cut the previous 2.59-second record by about 23%. A later 1.828-second claim (Fulcrum/Fable) is unacknowledged and has specification-gaming caveats; The two recent contributions are not outstandingly large by historical standards

Hutter Prize compression: too sparse

A prize for compressing a fixed 1 GB Wikipedia dump, scored on compressed size plus the decompressor, under a single-CPU time and memory cap (no GPUs). The corpus has been frozen since 2006; No entry claims a language model wrote the compressor. The leaderboard has not moved since October 2023

Gurobi mixed-integer programming: no acceleration

Gurobi reruns each new solver release on the same machine and the same model set, so the series is a vendor-reported speedup with hardware held fixed; Four annual releases (2022–2025) gained 13%, 8.6%, 13.1%, and 0.6%, a cumulative factor of about 1.4 since version 9.5. None of the announcements credits AI; The release is only annual, so we may see big speedups in 2026; The current pace of improvement seems to be slower than previously. Grace (2013) says “some mixed integer programming (MIP) algorithms, run on modern MIP instances with modern hardware, have roughly doubled in speed each year”

MIPLIB solution frontier: no acceleration

MIPLIB is a standard public library of mixed-integer programs. Maintainers accept improved solutions to open instances and periodically publish a new solution file, so the series is a history of public frontier updates rather than a timestamped lab notebook; Across 28 releases, the log reports hundreds of better incumbents plus a smaller number of first feasible solutions and optimality proofs. Submitters are not labeled consistently enough to count AI contributions; none are stated; The largest waves were in late 2019 and early 2020. Later years remain active but do not show a rising rate

Stockfish Elo: no acceleration

Stockfish is an open-source chess engine. A third party plays every development build 20,000 times against a frozen opponent (Stockfish 15), using one machine and one time control, so the series is pure software progress on a fixed Elo scale; About 675 Elo of software gain from 2013 to 2026, averaging around 50 Elo a year. Exactly one master commit credits a language model (a 0.6% speed patch in July 2026, not an Elo record); no Elo gain in the series is AI-attributed; There is no sign of acceleration

Matrix-multiplication exponent ω: no acceleration

The exponent $\omega$ is the best proved asymptotic upper bound on the number of operations needed to multiply two $n \times n$ matrices; The record fell from 2.8074 in 1969 to 2.3755 in 1990. The six improvements after 2010 moved it only 0.0024, from 2.3737 to 2.371339 in 2024; Every step is a human proof, and there is no sign of acceleration. AI has improved algorithms for finite matrix sizes, but not for the asymptotic exponent

“despite having basically every known fact about the world memorized, these models haven’t, as far as I know, made a single new discovery”

One possible explanation is the shift from LLMs trained on human-generated data to LLMs trained directly against the real world (RLVR). See the discussion in this January 2026 post on knowledge-creating versus knowledge-sharing LLMs.

When we do have a single number (either a set of problems, an upper or lower bound, or a measure of efficiency), it is generally based on problems with cheap validation.

In practice these are difficult to interpret for many reasons: problem selection and significance, contamination, inference-time scaling, and the division of work between models, scaffolds, and human experts.

One extreme view by a mathematician, Bartosz Naskręcki in August 2026: “At this pace we should expect about 1000+ conjectures proved every month and growing faster with each cycle … we should end this year with a pile of at least a million proven conjectures and solved open problems. The end of 2026 will mark a significant moment in human history where we will have almost all, say 99% of the current questions completely solved (new will be asked but I fix the bar) and not so many of them actually understood by humans at all.” ↩; Google DeepMind’s May 2025 AlphaEvolve announcement said: “By finding smarter ways to divide a large matrix multiplication operation into more manageable subproblems, it sped up this vital kernel in Gemini’s architecture by 23%, leading to a 1% reduction in Gemini’s training time.” OpenAI’s GPT-5.6 post said: “GPT‑5.6 Sol launched and monitored the speculator training process … [t]he resulting improvements increased token-generation efficiency by more than 15%.” ↩; E.g. Anthropic announced in April that it is donating up to $100M worth of Mythos Preview credits for discovering and fixing vulnerabilities; other labs have similar programs. ↩; Anthropic’s Fable 5 system card said: “we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).” Originally the fallback was silent, subsequently it became explicit. ↩; Microsoft wrote in July 2026: “As AI helps defenders discover more issues, customers will see a higher volume of security updates included in each security release.” See Evolving Windows vulnerability management to meet the speed of AI-powered discovery. ↩; Thomas Bloom’s erdosproblems.com says: “This is not intended as a record of when the solutions actually happened, or how quickly problems are being solved - they may have been solved weeks, months, or decades earlier, but it took time to verify or just be notified about the solution. This is just when their status was changed on this site, and has no meaning beyond that.” ↩; Parker Whitfill estimated a 700X compute efficiency gain over 2019-2026. ↩

Want to contribute to this work? METR is hiring: View open roles

Featured research

METR researches, develops and runs cutting-edge tests of AI capabilities, including broad autonomous capabilities and the ability of AI systems to conduct AI R&D.

A survey of 349 technical workers finds a median 1.4–2x self-reported change in value of work due to AI tools, expected to grow over time, though there are reasons to be skeptical of the magnitude.

Early Work on Monitorability Evaluations

We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring.

How Does Time Horizon Vary Across Domains?

We build on our time-horizon work and analyze 9 benchmarks for scientific reasoning, math, robotics, computer use, and self-driving in terms of time-horizon trends; we observe generally similar rates of improvement to the 7-month doubling time in our original time-horizon work.