ARC Prize's measured results for Mercury 3 Preview reasoning tiers and token consumption
On the ARC-AGI-1 and ARC-AGI-2 tasks, ARC Prize observed that the low and medium reasoning tiers of Mercury 3 Preview consume similar numbers of reasoning tokens. On the ARC-AGI-1 public tasks, the two tiers each used an average of 31k reasoning tokens per task, with scores of 89.50% and 89.25%, respectively. ARC Prize previously reported that Mercury 3 Preview scored 61.8% on ARC-AGI-2 at a cost of $0.06 per task, and scored 88.5% on ARC-AGI-1 at a cost of $0.03 per task.
Why read this
AI score 53
ARC Prize's measured results for Mercury 3 Preview show that the low and medium reasoning tiers consume similar amounts of reasoning tokens and achieve nearly identical scores, which can be used to understand the relationship between reasoning tiers and token consumption.
Topics
Body
@_inception_ai We also observed that on ARC-AGI-1 and ARC-AGI-2 tasks, low and medium reasoning used a similar number of reasoning tokens. On ARC-AGI-1 public tasks, for example, both averaged 31k reasoning tokens per task and scored 89.50% and 89.25%, respectively. https://t.co/L4LIjveyHK
AI Trends