Understand this story
Ai2 released the "Old Ideas, Novel Problems" study along with a supporting benchmark, data, and code, testing the reliability of six models—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—in judging the novelty of research ideas. The results show that small changes to the evaluation setup produce drastic differences: after switching GPT-5.4 to a simpler prompt on the Human+Generated data, its accuracy dropped from 93.5% to 40.9%, below the random level. The study also found that retrieval and additional reasoning do not reliably improve judgment quality; two purpose-built novelty assessment systems performed worse than a simple Claude Opus 4.6 prompt baseline and cost more.
AI-generated from public reports
Latest development · Report
Ai2 Releases "Old Ideas, Novel Problems" Study: LLMs Are Unreliable at Judging the Novelty of Research IdeasDevelopments
1 confirmed developments
- Source:Ai2 Blog
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Ai2 BlogFirst-partySelected
Ai2 Releases "Old Ideas, Novel Problems" Study: LLMs Are Unreliable at Judging the Novelty of Research Ideas
Ai2 released the "Old Ideas, Novel Problems" study and a companion benchmark, testing the reliability of six models—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—at judging the novelty of research ideas. The results show that small changes to the evaluation setup produce drastic differences: after switching GPT-5.4 to a simpler prompt on the Human+Generated data, its accuracy dropped from 93.5% to 40.9%, below chance level. The study also found that retrieval and additional reasoning do not reliably improve judgment quality; two purpose-built novelty evaluation systems performed worse than a simple Claude Opus 4.6 prompt baseline and cost more.