Harvard study says AI review comments comprehensively outperform high-level human peer review

On October 22, two researchers from Harvard University published a paper titled "AI Review Comments Comprehensively Outperform High-Level Human Peer Review," comparing 134 pairs of review comments written by AI and humans. The result: AI led across the board on seven evaluation dimensions.

The researchers also invited three experts who had previously served as editors-in-chief of major academic journals to conduct blind evaluations. In 30 comparisons, they judged the AI's review comments to be better 26 times.

Among the specific claims that were fact-checked, the AI review comments had a factual error rate of only 5.5%, while the human rate was as high as 15.9%.

Just last month, Terence Tao and 24 other Fields Medalists jointly criticized AI companies for rushing to announce mathematical breakthroughs, emphasizing that mathematical research requires rigorous verification and review.

The study used 124 academic papers and 134 human review comments provided by the rapidPeer service under Harvard University's Institute for Quantitative Social Science—a paid professional review service whose reviewers are selected by field experts, come from different research institutions, and are well compensated.

Their performance was originally quite good as well: among the 74 paper authors surveyed, 88% considered the review comments they received to be of high quality, and 60% gave the highest rating.

The researchers used Anthropic's Claude Opus 4.8 to generate review comments, then asked OpenAI's GPT-5.5 to serve as the AI judge, comparing the quality of human and AI reviews without knowing who wrote which comments.

The result was that humans lost thoroughly.

Even when the AI was not allowed to access the internet and was given only the same review guidelines as humans, GPT-5.5 chose the AI in all 134 comparisons.

After switching to specially optimized prompts and allowing internet access, 132 of the 134 comparisons clearly judged the AI as the winner, with the remaining two inconclusive; humans still did not achieve a single victory.

Specifically, on the three dimensions of supplementing references, identifying weaknesses in the paper, and research methods, the AI's win rate reached 100%; on theoretical innovation, argumentative framework, and quality of expression, its win rates also exceeded 90%.

The researchers even checked the 651 references with DOI or arXiv numbers cited by the AI and found that all of them actually existed, with not a single one fabricated.

So they invited the three former academic journal editors-in-chief mentioned at the beginning to each conduct 10 blind evaluations. As a result, in 30 comparisons, the human judges chose the AI 26 times.

On October 23, after the paper was released, Xu Yiqing, an associate professor of political science at Stanford University, publicly stated that academia should reconsider the role of human experts in peer review. In his view, human experts still need to participate, but there is no longer any need for them to personally write lengthy review comments of 1,500 words or more.

The answer may be very simple—it simply does not get tired.

Peer review is quite a taxing job. Reviewers need to read the paper thoroughly, check the research methods, data, charts, and arguments, and also look for relevant literature that the authors may have missed.

When introducing the experiment, the paper's authors even took the opportunity to complain about some reviewers at traditional journals: some start looking for fatal flaws sufficient to reject the paper by the third page, so that they do not have to keep reading.

In this experiment, the researchers set Claude Opus 4.8's reasoning intensity to the highest level and also specially designed a set of review prompts, allowing it to search online for materials.

However, the effect of this diligence was also quite obvious. Without internet access and using ordinary review guidelines, the AI's win rate in supplementing references was 62%; after switching to specially optimized prompts and allowing internet access, the win rate rose directly to 100%.

Times have changed.

In April this year, the editorial team of the academic journal Organization Science also published a study analyzing more than 10,000 review comments received over the past few years, finding that review reports heavily using AI were actually harder to read and discussed a narrower range of issues.

The paper's authors gave an example: AI sometimes nitpicks to an absurd degree—even though it knows that 2+2=4 is correct, it still points out that the author did not define what the plus sign means.

Previously, paper authors needed to prove to human reviewers that their research was worth publishing; in the future, it may be human reviewers' turn to prove to academic journals why they are still worth paying for.

Mathematicians may be a bit embarrassed

On September 11, Terence Tao and the first group of 25 Fields Medalists jointly spoke out, criticizing AI companies for treating solving difficult mathematical problems as a model capability competition, rushing to announce major breakthroughs without leaving the mathematics community enough time to verify, understand, and discuss them.

On September 29, the Advisory Group on Mathematics and AI (AGMAI) also specifically issued guidelines emphasizing that AI research results already fully understood by human mathematicians should be published as papers in accordance with traditional academic norms and undergo journal peer review.

Mathematical research has a set of rules that have lasted for many years. Once a proof is written, it still has to go through expert checking and peer discussion before it can gradually become knowledge recognized by the academic community.

On October 7, an organization called the Association for Human Mathematics (AHM) publicly criticized OpenAI for releasing more than 700 mathematical research documents at once, arguing that this looked more like a show of strength than serious scholarship.

But by this point, the matter had become somewhat absurd.

This article comes from the WeChat public account "Letter AI," author: Yuan Xinyue, published by 36Kr with authorization.