Google DeepMind: Training models to write moral arguments makes them more decisive but does not improve judgment
A new Google DeepMind paper found that after training Gemma-4 4B with reinforcement learning to write moral arguments, the proportion of cases where the model changed its answer on ambiguous MoralChoice cases due to a swap in option order dropped from 14.3% to 6.6%, but the proportion of sycophantic agreement with users only changed from 45.3% to 44.0%, and accuracy on moral fable questions actually fell from 56.5% to 53.4%. The paper recommends that when evaluating advice or judgment tasks, consistency and correctness be tested separately.
Body
New Google DeepMind paper finds that training a model to write better moral arguments makes it more decisive, but no better at moral judgment.
LLMs are more and more often asked for moral advice. A model that keeps its answer when the options are reordered is often taken as a sign of sound judgment.
They trained Gemma-4 4B with reinforcement learning to argue dilemmas taken from classic novels. The model judged its own arguments in pairs, and the winners set the reward.
Answer flips when options were swapped fell from 14.3% to 6.6% on ambiguous MoralChoice cases. Yet siding with the user barely moved, from 45.3% to 44.0%, and accuracy on moral fables dropped from 56.5% to 53.4%.
If you evaluate models for advice or judgment tasks, test consistency and correctness separately, since a steadier model can still be wrong.