腾讯混元预告 ExplorationBench 更多内容

新研究：我们发布 ExplorationBench，一个用于衡量 AI 系统探索能力的基准。 科学发现始于已知问题的尽头：一个系统必须提出假设、设计实验，并从结果中学习。评估这一点很难。真正全新的答案无法被快速检验，而在熟悉的领域里，模型只需回忆它见过的东西即可。 为应对这一挑战，来自腾讯混元、复旦大学和清华大学的研究者构建了可验证的 Alien Worlds。其规则是可执行的，因此每个答案都会被精确检验；同时它们与熟悉的知识相冲突，因此仅靠回忆无法解决这些任务。 🔹 两个沙盒：AlienCode（31 项隐藏规则变更，70 个任务）和 AlienLogic（24 条修补后的推理规则，70 个定理） 🔹 一本有缺陷的手册、四轮自行设计的探针，以及每轮之后的闭卷测试 🔹 每个答案都由解释器或证明检查器评分，不使用 LLM 评判 我们在 10 个前沿 AI 系统中发现： 1️⃣ 获得反馈比独自思考更有效。没有任何一次 AlienCode 运行在起始时超过 15.7%；四轮之后最佳达到 89.0%，而同样的轮次在没有反馈的情况下仍停留在 0.5–11.0%。 2️⃣ 设计实验很重要。重放一个系统自己最好的探针会为它提供完全相同的证据，但在 AlienCode 中，10 个系统里有 9 个的表现不如它们自己选择探针时。 3️⃣ 知道一条规则不等于会使用它。即使每条所需规则都被正确陈述，任务也只有 73.4% 的情况下被解决。 4️⃣ 单一分数掩盖了很多信息。同一系统在相同预算下最终得分从 5.7% 到 79.0% 不等，而且排名在两个世界之间几乎无法迁移。 CL-bench 问的是模型能否从上下文中学习。ExplorationBench 问的是它们能否自己发现规则。 📄 论文：https://t.co/eamwrsJyxQ 🌐 网站与排行榜：https://t.co/gxSeoCfc2s 📝 博客：https://t.co/jyrwH7ZnQH 💻 代码（即将发布）：https://t.co/i7VlgErkcl

查看原文

New Research: We are releasing ExplorationBench, a benchmark for measuring how AI systems explore. Scientific discovery begins where known problems end: a system has to frame hypotheses, design experiments, and learn from the results. Evaluating this is hard. Genuinely new answers cannot be checked quickly, and in familiar domains a model can simply recall what it has seen. Addressing this challenge, researchers from Tencent Hy, Fudan University, and Tsinghua University built verifiable Alien Worlds. Their rules are executable, so every answer is checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. 🔹 Two sandboxes: AlienCode (31 hidden rule changes, 70 tasks) and AlienLogic (24 patched inference rules, 70 theorems) 🔹 A flawed manual, four rounds of self-designed probes, and closed-book tests after every round 🔹 Every answer graded by an interpreter or a proof checker, with no LLM judge What we found across 10 frontier AI systems: 1️⃣ Getting feedback is more effective than thinking alone. No AlienCode run starts above 15.7%; after four rounds the best reaches 89.0%, while the same turns without feedback stay at 0.5–11.0%. 2️⃣ Designing the experiments matters. Replaying a system's own best probes gives it exactly the same evidence, yet in AlienCode 9 of 10 systems do worse than when they chose the probes themselves. 3️⃣ Knowing a rule is not using it. Even when every required rule is stated correctly, tasks are solved only 73.4% of the time. 4️⃣ One score hides a lot. The same system under the same budget ended anywhere from 5.7% to 79.0%, and rankings barely transfer between the two worlds. CL-bench asked whether models can learn from context. ExplorationBench asks whether they can discover the rules themselves. 📄 Paper: https://t.co/eamwrsJyxQ 🌐 Website & leaderboard: https://t.co/gxSeoCfc2s 📝 Blog: https://t.co/jyrwH7ZnQH 💻 Code (coming soon): https://t.co/i7VlgErkcl