ARC Prize announces evaluation results of Mercury 3 Preview on ARC-AGI
ARC Prize announced the evaluation results of Inception's Mercury 3 Preview on ARC-AGI: ARC-AGI-1 score 88.5%, $0.03 per task; ARC-AGI-2 score 61.8%, $0.06 per task; ARC-AGI-3 score 0.2%, $299 per task (provider adapter) or 0.1%, $333 (standard). This is the first diffusion model evaluated by ARC Prize; under instant reasoning effort it averages 981 output tokens/second on the ARC-AGI-2 public dataset, with a peak of 1,743 tokens/second for a single call; with reasoning enabled it averages 500-600 tokens/second. ARC Prize also provided links to the leaderboard, reproduction method, and testing policy.
Why read this
SelectedAI score 60
ARC Prize announced the scores and inference costs of Inception's Mercury 3 Preview on the three ARC-AGI benchmarks, and provided a reproducible entry point.
Topics
Body
@_inception_ai - Leaderboard: https://t.co/x8U1ItNLRj - Reproduce the public results: https://t.co/vMqtds8Zet - Testing policy: https://t.co/pBoUqvXe4P - Full Mercury 3 Preview results: https://t.co/F4A7ENPd6Q
AI Trends