Harvard study: AI review comments comprehensively outperform high-level human peer reviewers across seven dimensions

Terence Tao embarrassed: Harvard study shows AI peer review beats experts. Alphabet List download the client for exclusive early access. October 24, 2026 14:02:56 from Beijing. Updated October 24, 2026 14:14:51

On October 22, two researchers from Harvard University published a paper titled "AI Review Comments Comprehensively Beat High-Level Human Peer Review," comparing 134 pairs of AI- and human-written review comments. The result: AI led across all seven evaluation dimensions.

The researchers also invited three experts who had previously served as editors-in-chief of important academic journals to conduct blind reviews. In 30 comparisons, they judged the AI's review comments better 26 times.

Among the specific claims that were checked, the proportion of factual errors in AI review comments was only 5.5%, while for humans it was as high as 15.9%.

Just last month, Terence Tao and 25 other Fields Medal winners jointly criticized AI companies for rushing to announce mathematical breakthroughs, stressing that mathematical research requires rigorous verification and review.

The study used 124 academic papers and 134 human review comments provided by the rapidPeer service under Harvard University's Institute for Quantitative Social Science—a paid professional review service whose reviewers are selected by field experts, come from different research institutions, and are well compensated.

Their performance was originally quite good as well: among the 74 paper authors surveyed, 88% considered the review comments they received to be of high quality, and 60% gave the highest-tier rating.

The researchers used Anthropic's Claude Opus 4.8 to generate review comments, then asked OpenAI's GPT-5.5 to serve as the AI judge, comparing the quality of human and AI reviews without knowing who wrote the comments.

The result: humans lost very thoroughly.

Even when the AI was not allowed to access the internet and was given only the same review guidelines as humans, in all 134 comparisons GPT-5.5 chose the AI.

After switching to specially optimized prompts and allowing internet access, 132 of the 134 comparisons clearly judged the AI the winner, the remaining two were inconclusive, and humans still did not achieve a single victory.

Specifically, on the three dimensions of supplementing references, identifying weaknesses in the paper, and research methods, the AI's win rate reached 100%; in theoretical innovation, argumentative framework, expression quality, and other aspects, the win rates also all exceeded 90%.

The researchers even checked the 651 references with DOI or arXiv numbers cited by the AI and found that all of them actually exist, with not a single one fabricated.

So they invited the three former academic journal editors-in-chief mentioned at the beginning, each conducting 10 blind reviews. As a result, in 30 comparisons, the human judges chose the AI 26 times.

On October 23, after the paper was released, Xu Yiqing, an associate professor of political science at Stanford University, publicly stated that academia should reconsider the role of human experts in peer review. In his view, human experts still need to participate, but there is no longer any need for them to personally write lengthy review comments of 1,500 words or more.

The answer may be very simple—it simply does not get tired.

Peer review is quite mentally taxing work. Reviewers need to read the paper thoroughly, check the research methods, data, charts, and arguments, and also look for relevant literature the authors may have missed.

When introducing the experiment, the paper's authors even took the opportunity to complain about some traditional journal reviewers: some start looking for fatal flaws sufficient to reject the paper by the third page, so that they do not have to keep reading.

In this experiment, the researchers set Claude Opus 4.8's reasoning intensity to the highest level and also specially designed a set of review prompts, allowing it to search online for materials.

However, the effect of this diligence was also quite obvious. Without internet access and using ordinary review guidelines, the AI's win rate in supplementing references was 62%; after switching to specially optimized prompts and allowing internet access, the win rate rose directly to 100%.

Times are different now.

In April this year, the editorial team of the academic journal Organization Science also published a study analyzing more than 10,000 review comments received over the past few years, finding that review reports heavily using AI were instead harder to read and discussed narrower issues.

The paper's authors gave an example: AI sometimes nitpicks to an absurd degree—clearly knowing that 2+2=4 is correct, yet still pointing out that the author did not define what the plus sign means.

Previously, paper authors needed to prove to human reviewers that their research was worth publishing; in the future, it may be human reviewers' turn to prove to academic journals why they are still worth paying for.

03

Mathematicians may be a bit embarrassed

On September 11, Terence Tao and the first batch of 25 Fields Medal winners jointly spoke out, criticizing AI companies for treating solving difficult mathematical problems as a model capability competition, rushing to announce major breakthroughs without leaving the mathematics community enough time to verify, understand, and discuss.

On September 29, the Advisory Group on Mathematics and AI (AGMAI) also specifically issued guidelines, stressing that AI research results already fully understood by human mathematicians should be published as papers in accordance with traditional academic norms and undergo journal peer review.

Mathematical research has a set of rules that has continued for many years. Once a proof is written, it still has to go through expert checking and peer discussion before it can gradually become knowledge recognized by the academic community.

On October 7, an organization called the Association for Human Mathematics (AHM) publicly criticized OpenAI for releasing more than 700 mathematical research documents at once, arguing that this looked more like a show of strength than serious scholarship.

But at this point in the story, it seems somewhat absurd.

"Special statement: The above content (including videos, pictures, or audio) is uploaded and published by users of 'Dafeng Hao,' a self-media platform under Phoenix Net. This platform only provides information storage space services.

Notice: The content above (including the videos, pictures and audios if any) is uploaded and posted by the user of Dafeng Hao, which is a social media platform and merely provides information storage space services."