AI Trends

Microsoft Paper Finds Large Language Models Abandon Correct Answers When Incorrectly Challenged

Rohan PaulUpdated Nov 19 05:08

AI summary

A Microsoft paper finds that large language models often abandon their originally correct answers when incorrectly challenged by users or other agents, and that additional accuracy training, sampling, or self-reflection does not solve the problem. After Self-Disagreement training, the proportion of Qwen3-8B and Gemma-2 9B models maintaining correct answers under incorrect challenges rose from 53.3% and 25.1%, respectively, to about 89%, while accuracy-only training did not produce this capability.

Why read this

AI score 72

The study reveals that accuracy training cannot address models changing their answers after incorrect challenges and provides a transferable targeted training method.

Topics

Tags

Body

New Microsoft paper finds that LLMs give up correct answers when challenged with wrong arguments, and accuracy training does not fix it.

When a user or another agent disputes an answer, models often abandon correct answers even when the challenge is wrong. More accuracy training, more sampling, or self-reflection does not fix it.

Self-Disagreement trains one model to state an answer, write the strongest case against it, then rule on that case and answer again. A standard answer checker rewards the final answer and the ruling.

When challengers were wrong, base Qwen3-8B kept its correct answer 53.3% of the time and Gemma-2 9B 25.1%. After training, both kept it about 89% of the time, while accuracy-only training added no such skill.

If your model faces pushback from users or other agents, test how often it drops correct answers, since accuracy scores will not show it.

Source

Rohan Paul