Understand this story
Ai2, in collaboration with Carnegie Mellon University, NVIDIA, and Johns Hopkins University, released the XYBench benchmark, designed to test whether AI assistants can identify erroneous assumptions in users' questions and guide them toward solving their true goals. The benchmark contains 8,115 real-world questions and won the EMNLP 2026 Best Resource Paper Award. On Stack Overflow questions, among the responses from Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, 75–86% contained solutions to the surface request, only 59–71% addressed the true goal, and only 35–41% of responses focused primarily on the true goal. When the same batch of models made binary choices, 71–84% were able to select the answer oriented toward the true goal, revealing a gap between recognizing useful advice and proactively offering it.
AI-generated from public reports
Developments
1 confirmed developments
- Source:Ai2 Blog
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 14
Ai2 BlogFirst-party
Ai2 Releases XYBench: Testing Whether AI Assistants Can Identify the False Assumptions Behind Users' Questions
Ai2, in collaboration with Carnegie Mellon University, NVIDIA, and Johns Hopkins University, released XYBench, which contains 8,115 real-world questions designed to test whether AI assistants can identify the false assumptions in users' questions and guide them toward solving their true goal. The benchmark won the Best Resource Paper Award at EMNLP 2026. On Stack Overflow questions, 75–86% of the responses from Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro contained solutions to the surface request, only 59–71% addressed the true goal, and only 35–41% of responses focused primarily on the true goal.
Ai2 BlogRelated discussion
Ai2 Blog joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
Hugging FaceFirst-party
XYBench: Helping Models See the Real Needs Behind Questions
Allen AI built the XYBench benchmark, using 1,557 Stack Overflow questions, 184 Stack Exchange questions, 102 manually curated examples, and 6,272 questions generated from WikiHow, to distinguish between the request a user explicitly states (Y) and the goal they actually want to achieve (X). On Stack Overflow, among the responses from Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, 75–86% contained a solution to the explicitly stated request, while only 59–71% addressed the real goal, and responses focused primarily on the real goal accounted for only 35–41%. In pairwise judgments, the same set of models could select the answer aimed at the real goal 71–84% of the time, indicating a gap between recognizing useful advice and proactively offering it.
Heat trend for this story
- Current
- 5
- Period peak
- 20 Nov 14 02:00
- 24-hour change
- -50%
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.