←Search results

Search “arXiv”

By relevance · Total 78 results Clear search

Sunday

Wednesday

  1. Ai2 BlogSelected

    Ai2 Releases "Old Ideas, Novel Problems" Study: LLMs Are Unreliable at Judging the Novelty of Research Ideas

    Ai2 released the "Old Ideas, Novel Problems" study and a companion benchmark, testing the reliability of six models—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—at judging the novelty of research ideas. The results show that small changes to the evaluation setup produce drastic differences: after switching GPT-5.4 to a simpler prompt on the Human+Generated data, its accuracy dropped from 93.5% to 40.9%, below chance level. The study also found that retrieval and additional reasoning do not reliably improve judgment quality; two purpose-built novelty evaluation systems performed worse than a simple Claude Opus 4.6 prompt baseline and cost more.

    Ai2 released the "Old Ideas, Novel Problems" study and a companion benchmark, testing the reliability of six models—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—at judging the novelty of research ideas. The results show that small changes to the evaluation setup produce drastic differences: after switching GPT-5.4 to a simpler prompt on the Human+Generated data, its accuracy dropped from 93.5% to 40.9%, below chance level. The study also found that retrieval

    Matched fieldsBody

    #lab#Papers/Research#Evaluations / benchmarks#Agents

Saturday

  1. TMTPost · AGI

    Harvard study says AI review comments comprehensively outperform high-level human peer reviewers

    Two researchers from Harvard University published a paper on October 22, comparing 134 pairs of AI and human review comments; AI led comprehensively across seven evaluation dimensions. Three experts who had served as editors-in-chief of major academic journals conducted 30 blind evaluations, and in 26 of them they judged the AI review comments to be better. The study used 124 papers and 134 paid professional review comments provided by rapidPeer, with Claude Opus 4.8 generating the reviews and GPT-5.5 serving as the blind-evaluation judge. The factual error rate of AI review comments was 5.5%, compared with 15.9% for humans.

    Two researchers from Harvard University published a paper on October 22, comparing 134 pairs of AI and human review comments; AI led comprehensively across seven evaluation dimensions. Three experts who had served as editors-in-chief of major academic journals conducted 30 blind evaluations, and in 26 of them they judged the AI review comments to be better. The study used 124 papers and 134 paid professional review comments provided by rapidPeer, with Claude Opus 4.8 generating the reviews and G

    Matched fieldsBody

    #hot-signal#Papers/Research#Evaluations / benchmarks#Anthropic

Friday

  1. Hugging Face

    Microsoft uses ThinkingBox to turn database scoring into RL rewards, fine-tuning Qwen3.8-27B

    The Microsoft Copilot Studio team, in collaboration with Toloka and others, directly uses the executable final-state database checks of ThinkingBox-Bench as reinforcement learning rewards to perform full-parameter RL fine-tuning on Qwen3.8-27B. On 507 out-of-distribution tasks not involved in training, the number of tasks the model passed in all 20 attempts rose from 38 (7.5%) to 107 (21.1%), pass@1 rose from 51.70% to 60.78%, and the cost per successful attempt dropped from $0.177 to $0.139. The training pool was filtered using offline-policy relative difficulty, ultimately retaining 187 tasks; the training and evaluation code has been open-sourced at microsoft/thinkingbox-training.

    The Microsoft Copilot Studio team, in collaboration with Toloka and others, directly uses the executable final-state database checks of ThinkingBox-Bench as reinforcement learning rewards to perform full-parameter RL fine-tuning on Qwen3.8-27B. On 507 out-of-distribution tasks not involved in training, the number of tasks the model passed in all 20 attempts rose from 38 (7.5%) to 107 (21.1%), pass@1 rose from 51.70% to 60.78%, and the cost per successful attempt dropped from $0.177 to $0.139. Th

    Matched fieldsBody

    #lab#Papers/Research#Agents#English text is being prepared.

Thursday

  1. Unite.AI

    New paper proposes BackUpper: making coding agents back up before they act

    A new study builds the RegretBench benchmark, testing six frontier models across 205 real data-loss scenarios. Results show agents complete tasks at an average rate of 86.1%, but back up before critical operations only 15.2% of the time—GPT-5.6-Terra at 4.4%, GPT-5.6-Sol at 5.9%, Gemini 3.7-Flash at 2.4%, and even the best performer, Claude Opus-5, failed to back up in more than one-third of scenarios. The authors' proposed BackUpper runs in parallel with the agent as an external plugin, using a four-tier strategy: replacing destructive commands, pairing backup commands, generating snapshots, or calling the GPT-5.6-Luna sub-model to determine which data needs saving. In testing it covered all critical operations, with recovered data averaging 92.2% similarity to the original state, at an additional cost of $0.014 per task. BackUpper already supports Claude Code, Codex, Hermes, OpenHands, DeepSeek Harness, and OpenClaw, and its code and dataset have been open-sourced.

    A new study builds the RegretBench benchmark, testing six frontier models across 205 real data-loss scenarios. Results show agents complete tasks at an average rate of 86.1%, but back up before critical operations only 15.2% of the time—GPT-5.6-Terra at 4.4%, GPT-5.6-Sol at 5.9%, Gemini 3.7-Flash at 2.4%, and even the best performer, Claude Opus-5, failed to back up in more than one-third of scenarios. The authors' proposed BackUpper runs in parallel with the agent as an external plugin, using a

    Matched fieldsBody

    #hot-signal#Papers/Research#Agents#Safety / alignment

Monday

Thursday

Wednesday

Tuesday