Stanford paper finds that all 8 LLMs prefer their own medical answers in anonymous peer review
A new Stanford paper finds that all 8 LLMs tested rank their own answers higher in anonymous peer review than other models rank them. The study had 8 models from OpenAI, Anthropic, Google, and Qwen answer 620 real clinical questions, then rank the 8 anonymized answers; the average self-preference was 1.22 places, with GPT-5.6 Sol at nearly 3 places. The preference persisted after removing the scoring rubric, and disclosing model names only reduced the preference by 0.094 places. The authors recommend that when using LLM judges, one should adopt judging panels spanning multiple model families and report results per judge rather than as a single aggregate score.
Why read this
AI score 64
A Stanford paper used 620 real clinical questions for anonymous peer review, quantifying the magnitude of LLM judges' preference for their own answers, offering direct reference for those building evaluation pipelines.
Topics
Body
New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous.
Medical AI benchmarks often use LLMs to grade other LLMs' answers. If a judge favors its own outputs, the benchmark tilts toward that model.
They had 8 models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all 8 anonymous answers. They compared where each judge placed its own answer with where the other 7 judges placed that same answer.
Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias held without a rubric, and revealing model names cut it by only 0.094 positions.
If you use LLM judges, use a panel from several model families and report results per judge instead of 1 pooled score.
AI Trends