Chelsea Finn：ScholarCatalyst 评测 LLM 科研过程
Scientific discovery is one of the most exciting applications of LLMs, but we lack data & evals on the scientific process.
科学发现是 LLM 最令人兴奋的应用之一，但我们缺乏关于科研过程的数据与评测。 ScholarCatalyst 捕捉的是哪些先前论文能推进某个问题的新研究，而非引用或主题相关性。 引人注目的是，即便是前沿模型也难以识别哪些先前研究有帮助——未来工作空间很大！
查看原帖AI can increasingly make progress on open problems like Navier-Stokes. Yet defining a new problem still relies on human research taste. Introducing ScholarCatalyst, a benchmark built from AI researchers’ firsthand accounts of what inspired their work.
正文
当前显示原文，中文全文译文尚未就绪。
Scientific discovery is one of the most exciting applications of LLMs, but we lack data & evals on the scientific process.
ScholarCatalyst captures which prior papers can advance new research on a problem, rather than citations or topical relevance.
Strikingly, even frontier models struggle to identify which prior research is helpful -- lots of room for future work!