XYBench: Helping Models See the Real Needs Behind Questions
Allen AI built the XYBench benchmark, using 1,557 Stack Overflow questions, 184 Stack Exchange questions, 102 manually curated examples, and 6,272 questions generated from WikiHow, to distinguish between the request a user explicitly states (Y) and the goal they actually want to achieve (X). On Stack Overflow, among the responses from Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, 75–86% contained a solution to the explicitly stated request, while only 59–71% addressed the real goal, and responses focused primarily on the real goal accounted for only 35–41%. In pairwise judgments, the same set of models could select the answer aimed at the real goal 71–84% of the time, indicating a gap between recognizing useful advice and proactively offering it.
Body
Building XYBench
We built XYBench from four sources: 1,557 questions from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 questions generated from WikiHow guides. For each question, we identified the stated request (Y), what the person explicitly asks for, and the intended request (X)—the underlying goal that can be inferred from context.
For the public forum questions, we used the three highest-scored human answers and the accepted answer, where available, to establish the intended request and possible solutions. We filtered out cases where answering the stated request was judged sufficient. For WikiHow, we used Sonnet 4.5 to generate questions around plausible misconceptions in procedural guides, then refined them to include enough context for an expert to infer the underlying goal without stating it outright. We removed examples where answering the literal request would largely resolve the problem.
Evaluating how an answer helps
A model might spend several paragraphs recommending a more powerful vacuum cleaner, then briefly suggest checking for a clogged hose. If a clog is causing the weak suction, clearing it could solve the problem.
A more helpful answer would recognize that the user wants to restore suction and focus on diagnosing the cause. This draws on pragmatics, the branch of linguistics concerned with how context and communicative intent shape meaning. To test whether models use this understanding in their answers, XYBench measures three aspects of each response:
Pragmatic solution —records whether a model’s response includes solutions to the stated request (Y) and the intended request (X), checking each separately.
Pragmatic emphasis —measures whether a model’s answer focuses on the stated request (Y), focuses on the intended request (X), or balances the two.
Misconception identification —measures whether a model’s response steers the user toward the underlying goal (X) while discouraging or avoiding the mistaken approach (Y).
In XYBench, Claude Sonnet 4.5 serves as the automated judge. It classifies individual suggestions within each model’s response and assesses the answer’s pragmatic emphasis.
To validate Sonnet 4.5’s judgments, two human reviewers independently assessed 500 model responses to XYBench questions. Agreement with the judge, averaged across the reviewers, ranged from 97.5–100% on whether a solution appeared, 84.6–93.3% on misconception identification, and 74.2–79.8% on pragmatic emphasis.
What the experiments show
We evaluated Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro on XYBench in a single-turn setting, with one response per question. We evaluated human reference answers separately using the same metrics.
Across all four sources, every model was more likely to include a solution to the stated request (Y) than to the underlying goal (X). On Stack Overflow, stated-request solutions appeared in 75–86% of responses, compared with 59–71% for underlying-goal solutions.
A passing mention of a solution does not make it the focus of an answer. In the vacuum example, a model could briefly suggest unclogging the hose but focus most of its response on buying a more powerful vacuum. We would count that response as containing a solution to the underlying goal, even though the model emphasized the stated request. We measured overall emphasis separately: on Stack Overflow, models focused primarily on the underlying goal in only 35–41% of their responses, and on the stated request in 47–57%. In the remaining responses, models gave similar emphasis to the stated request and the underlying goal.
We also tested whether making the user’s goal explicit would help. In an oracle experiment, we supplied both the stated request (Y) and the intended request (X) alongside each question. On Stack Overflow, Sonnet 4.6’s rate of including a solution to the underlying goal rose from 59% to 67%, but its misconception-identification rate fell from 47% to 34%. Across the sources with human reference answers, every model in this experiment still trailed humans on misconception identification. Supplying intent helped, but did not consistently close the gap with human responses. This suggests that detecting intent is only part of the challenge: models also need to act on it when it diverges from the stated request.
A separate experiment tested whether models could identify useful advice when judging responses instead of writing them. We gave the same five models, plus GPT-4.1, each question with two possible answers: one focused on the literal request and one explaining the misconception and addressing the underlying goal. On Stack Overflow, all six chose the goal-focused answer in 71–84% of cases. By comparison, only 35–41% of the answers written by the five models in our main evaluation primarily focused on the underlying goal. The results expose a gap between recognizing useful advice and producing it, which a multiple-choice test alone would miss.
What this means for evaluation
Researchers can use XYBench to test whether changes to prompting or training help models identify misconceptions, offer solutions to users’ underlying problems, and make those solutions the focus of their answers. Measuring each behavior individually lets researchers see whether progress in one also improves the others.
Future work could test which training methods help models act on inferred intent—AKA what users need and not just what they literally asked for. The XYBench currently covers English-language queries and mainly procedural problems; extending it to other languages, cultures, and conversational settings remains an open direction.
We hope XYBench enables researchers to build models that offer useful advice even when a user’s first question rests on a mistaken assumption.
Datasets mentioned in this article 1
allenai/XYBench-dataset
More from this author
ToyBlocks: A controlled world for robot learning and evaluation
StateBench: Testing how language models track state changes over long documents
Datasets mentioned in this article 1
allenai/XYBench-dataset