Harvard study says AI review comments comprehensively outperform high-level human peer reviewers

On October 22, two researchers from Harvard University published a paper titled "AI Review Comments Comprehensively Outperform High-Level Human Peer Review," comparing 134 pairs of review comments written by AI and humans. The result: AI led across all seven evaluation dimensions.

The researchers also invited three experts who had previously served as editors-in-chief of major academic journals to conduct blind reviews. In 30 comparisons, they judged the AI's review comments to be better 26 times.

Among the specific claims that were fact-checked, the proportion of factual errors in AI review comments was only 5.5%, while for humans it was as high as 15.9%.

Just last month, Terence Tao and 24 other Fields Medalists jointly criticized AI companies for rushing to announce mathematical breakthroughs, stressing that mathematical research requires rigorous verification and review.

Humans Crushed

The reviewers who suffered this crushing defeat were no randomly recruited volunteers.

The study used 124 academic papers and 134 human review comments provided by the rapidPeer service under Harvard University's Institute for Quantitative Social Science—a paid professional review service whose reviewers are selected by field experts, come from different research institutions, and are well compensated.

Their performance was originally quite good as well: among the 74 paper authors surveyed, 88% considered the review comments they received to be of high quality, and 60% gave the highest rating.

The researchers used Anthropic's Claude Opus 4.8 to generate review comments, then asked OpenAI's GPT-5.5 to serve as the AI judge, comparing the quality of human and AI reviews without knowing who had written each comment.

The result: humans lost thoroughly.

Even when the AI was not allowed to access the internet and was given only the same review guidelines as humans, GPT-5.5 chose the AI in all 134 comparisons.

After switching to specially optimized prompts and allowing internet access, 132 of the 134 comparisons clearly judged the AI as the winner, with the remaining two inconclusive—humans still did not win a single time.

Specifically, in the three dimensions of supplementing references, identifying weaknesses in the paper, and research methods, the AI's win rate reached 100%; in theoretical innovation, argumentative framework, and quality of expression, its win rate also exceeded 90%.

The researchers even checked the 651 references with DOI or arXiv numbers cited by the AI and found that all of them genuinely existed—not a single one was fabricated.

So they brought in the three former academic journal editors-in-chief mentioned at the beginning, each conducting 10 blind reviews. In the end, in 30 comparisons, the human judges chose the AI 26 times.

On October 23, after the paper was released, Xu Yiqing, an associate professor of political science at Stanford University, publicly stated that academia should reconsider the role of human experts in peer review. In his view, human experts still need to participate, but there is no longer any need for them to personally write lengthy review comments of 1,500 words or more.

The answer may be very simple—it simply never gets tired.

Peer review is quite mentally taxing work. Reviewers need to read the paper thoroughly, check the research methods, data, charts, and arguments, and also look for relevant literature the authors may have missed.

When introducing the experiment, the paper's authors even took the opportunity to poke fun at some traditional journal reviewers: some start looking for fatal flaws sufficient to reject the paper by the third page, so that they won't have to keep reading.

In this experiment, the researchers set Claude Opus 4.8's reasoning intensity to the maximum and also specially designed a set of review prompts, allowing it to search online for materials.

However, the effect of this diligence was quite obvious. Without internet access and using ordinary review guidelines, the AI's win rate in supplementing references was 62%; after switching to specially optimized prompts and allowing internet access, the win rate rose directly to 100%.

Times have changed.

In April this year, the editorial team of the academic journal Organization Science also published a study analyzing more than 10,000 review comments received over the past few years, finding that review reports heavily using AI were actually harder to read and discussed a narrower range of issues.

The paper's authors gave an example: sometimes the AI even nitpicks to an absurd degree—clearly knowing that 2+2=4 is correct, yet still pointing out that the author did not define what the plus sign means.

Previously, paper authors needed to prove to human reviewers that their research was worth publishing; in the future, it may be human reviewers' turn to prove to academic journals why they are still worth paying for.

Mathematicians may be feeling a bit awkward

On September 11, Terence Tao and the first batch of 25 Fields Medalists jointly spoke out, criticizing AI companies for treating cracking mathematical problems as a model capability competition, rushing to announce major breakthroughs without leaving the mathematics community enough time to verify, understand, and discuss them.

On September 29, the Advisory Group on Mathematics and AI (AGMAI) issued special guidelines emphasizing that AI research results already fully understood by human mathematicians should be published as papers in accordance with traditional academic norms and undergo journal peer review.

Mathematical research has a set of rules that have lasted for many years. Once a proof is written, it still has to go through expert checking and peer discussion before it can gradually become knowledge recognized by the academic community.

On October 7, an organization called the Association for Human Mathematics (AHM) publicly criticized OpenAI for releasing more than 700 mathematical research documents at once, arguing that this looked more like a show of strength than serious scholarship.

But by this point in the story, it all seems somewhat absurd.