Understand this story
Researchers from Microsoft and City University of Hong Kong have built Taste-Bench to evaluate the long-term decision-making capabilities of AI agents on long tasks. The benchmark contains 502 binary-choice questions. According to The Decoder, the best-performing model, GPT-5.6 Sol, achieved an average accuracy of only 59.7%, while GPT-5.5 reached 59.5%. When key clues only become apparent after a large amount of subsequent work, the average accuracy of all models drops to 21%, close to random guessing, and increasing the reasoning budget brings almost no improvement. The researchers also distilled the teacher model's judgments using Qwen3.6-27B as the base model; the student's accuracy on unseen tasks rose from 30% to 47.9%, and its success rate on 41 SWE-bench Pro tasks increased from 14.6% to 33.7%.
AI-generated from public reports
Developments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 18
The Decoder
Microsoft and CityU Hong Kong propose Taste-Bench to evaluate AI agents' long-term decision-making capabilities
Researchers from Microsoft and City University of Hong Kong propose Taste-Bench, using 502 binary-choice questions to evaluate AI agents' ability to make long-term decisions in long-horizon tasks. The best-performing model, GPT-5.6 Sol, achieves only 59.7% average accuracy, while GPT-5.5 reaches 59.5%. When key clues only become apparent after substantial subsequent work, the average accuracy of all models drops to 21%, close to random guessing, and increasing the reasoning budget brings almost no improvement. The researchers also distilled the teacher model's judgments using Qwen3.6-27B as the base model; the student's accuracy on unseen tasks rose from 30% to 47.9%, and its success rate on 41 SWE-bench Pro tasks increased from 14.6% to 33.7%.