AI Trends

ARC Prize announces evaluation results of Mercury 3 Preview on ARC-AGI

ARC PrizeUpdated Nov 17 23:22

AI summary

ARC Prize announced the evaluation results of Inception AI's Mercury 3 Preview on ARC-AGI: a score of 61.8% on ARC-AGI-2 at $0.06 per task, 88.5% on ARC-AGI-1 at $0.03 per task, and only 0.2% on ARC-AGI-3. In the instant reasoning tier, no reasoning tokens are used; on the public ARC-AGI-2 dataset, the model averages 981 tokens/s of output, with a peak of 1,743 tokens/s in a single call (3,001 tokens output in 1.72 seconds); with reasoning enabled, it averages 500-600 tokens/s. ARC Prize states that this is the first diffusion model it has evaluated.

Why read this

SelectedAI score 67

ARC Prize announced the scores and per-task costs of Mercury 3 Preview across ARC-AGI versions, and presented the throughput differences under different reasoning tiers.

Topics

Tags

Body

@_inception_ai With instant reasoning effort (which used no reasoning tokens), Mercury 3 Preview averaged 981 output tokens/second on the ARC-AGI-2 public dataset, peaking at 1,743 tokens/second on a single call (3,001 tokens in 1.72 seconds). https://t.co/xCdOMAz6sO

Source

ARC Prize