Understand this story
On October 30, 2026, Elorian and Mercor jointly released the DecoyCount benchmark to test whether models are truly counting objects or relying on visual shortcuts. According to Latent Space, frontier models all scored below 50% accuracy on this benchmark, with the highest being GPT-6 Astra at 43.17%. Elorian said its visual reasoning model Glimpse scored 50.48% in a single run on 1,660 questions, making it the only model to exceed 50%, but that result came from internal testing and did not use an agent or tool framework, whereas the six models in the paper each took the average of three runs. Elorian CEO Andrew Dai said the benchmark examines both vision and reasoning, and can distinguish whether a model failed to see the objects or miscounted them.
AI-generated from public reports
Latest development · Report
Elorian Releases DecoyCount Benchmark: Frontier Models All Score Below 50% on Visual Counting AccuracyDevelopments
1 confirmed developments
- Source:Latent Space
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 30
Latent Space
Elorian Releases DecoyCount Benchmark: Frontier Models All Score Below 50% on Visual Counting Accuracy
Elorian and Mercor jointly released the DecoyCount benchmark, which tests whether models truly count objects or rely on visual shortcuts. Frontier models all scored below 50% accuracy, with the highest being GPT-6 Astra at 43.17%. Elorian says its visual reasoning model Glimpse scored 50.48% in a single run on 1,660 questions, making it the only model to exceed 50%, but this result comes from internal testing and did not use an agent or tool framework, whereas the six models in the paper each took the average of three runs. Elorian CEO Andrew Dai said the benchmark examines both vision and reasoning, and can distinguish whether a model failed to see the objects or miscounted them.