ARC Prize's hands-on test of Inception AI's Mercury 3 Preview…
ARC Prize tested Inception AI's Mercury 3 Preview on the ARC-AGI benchmark: it scored 61.8% on ARC-AGI-2 at $0.06 per task, 88.5% on ARC-AGI-1 at $0.03 per task, and only 0.2% on ARC-AGI-3. With the instant reasoning tier enabled, the model averaged 981 tokens/second output on the public ARC-AGI-2 dataset, with a peak of 1,743 tokens/second in a single call (3,001 tokens output in 1.72 seconds); with reasoning enabled, it averaged 500-600 tokens/second. ARC Prize says this is the first diffusion model it has evaluated.
Why read this
AI score 55
ARC Prize's hands-on test of Mercury 3 Preview provides scores and throughput data across reasoning tiers, allowing comparison of its speed versus accuracy trade-offs.
Topics
Body
@_inception_ai With reasoning enabled, Mercury 3 Preview averaged 500-600 tokens per second: https://t.co/1bHNxkjFZh
AI Trends