AI 热点

斯坦福论文发现 8 个 LLM 在匿名互评中均偏好自己的医疗答案

Rohan Paul更新于 11月18日 13:49

New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous.

AI 导读

斯坦福新论文发现,测试的 8 个 LLM 在匿名互评时都给自己回答的排名高于其他模型给它的排名。研究让来自 OpenAI、Anthropic、Google 和 Qwen 的 8 个模型回答 620 个真实临床问题,再对 8 份匿名答案排序,平均自我偏好为 1.22 个名次,GPT-5.6 Sol 接近 3 个名次。去掉评分细则后偏好依然存在,公开模型名称只让偏好减少 0.094 个名次。作者建议使用 LLM 评审时采用跨模型家族的评审小组,并按评审分别报告结果而非单一汇总分。

推荐理由

AI 评分 64

斯坦福论文用 620 道真实临床问题做匿名互评,量化了 LLM 评审对自身答案的偏好幅度,对搭建评测流程的人有直接参考。

主题

标签

正文

New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous.

Medical AI benchmarks often use LLMs to grade other LLMs' answers. If a judge favors its own outputs, the benchmark tilts toward that model.

They had 8 models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all 8 anonymous answers. They compared where each judge placed its own answer with where the other 7 judges placed that same answer.

Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias held without a rubric, and revealing model names cut it by only 0.094 positions.

If you use LLM judges, use a panel from several model families and report results per judge instead of 1 pooled score.

来源

Rohan Paul