University of Chicago: LLMs' self-reported key decision factors are unreliable
A new paper from the University of Chicago finds that LLMs cannot predict the key issues behind disagreements in their own answers; the key decision factors they self-report match the true factors only 13.4% of the time, lower than the 39.2% for randomly shuffled ones. The study had models repeatedly predict 59 March Madness games and 15 Metaculus events, then used a reranker to compare answer pairs, requiring about 86 times less computation than local LLM judging. A single answer covered 28.6% of the true key factors, versus a baseline of 37.8%.
Body
New University of Chicago paper suggests that an LLM's own account of what matters to its decision is a poor guide to what actually varies across its answers.
That LLMs fail to predict the key questions their own answers split on, which only comparing many answers reveals.
When an LLM recommends something, you want its cruxes, the few sub-questions that drove its conclusion. Asking the model upfront, or reading 1 of its answers, is cheap but may miss what really drives it.
They had models forecast 59 March Madness games and 15 Metaculus events many times, then compared pairs of forecasts to find disputed facts. A reranker made that matching cheap, at an estimated 86 times less compute than a local LLM judge.
In March Madness, predicted cruxes shared 13.4% of clusters with the real ones, below the 39.2% a random shuffle gives. A single answer did better but covered only 28.6% of real cruxes, against a 37.8% baseline.
For high-stakes decisions, sample many answers and look where they disagree, instead of trusting the model's own list of what matters.