AI Trends

ARC Prize's measured results for Mercury 3 Preview reasoning tiers and token consumption

ARC PrizeUpdated Nov 17 23:22

AI summary

On the ARC-AGI-1 and ARC-AGI-2 tasks, ARC Prize observed that the low and medium reasoning tiers of Mercury 3 Preview consume similar numbers of reasoning tokens. On the ARC-AGI-1 public tasks, the two tiers each used an average of 31k reasoning tokens per task, with scores of 89.50% and 89.25%, respectively. ARC Prize previously reported that Mercury 3 Preview scored 61.8% on ARC-AGI-2 at a cost of $0.06 per task, and scored 88.5% on ARC-AGI-1 at a cost of $0.03 per task.

Why read this

AI score 53

ARC Prize's measured results for Mercury 3 Preview show that the low and medium reasoning tiers consume similar amounts of reasoning tokens and achieve nearly identical scores, which can be used to understand the relationship between reasoning tiers and token consumption.

Topics

Tags

Body

@_inception_ai We also observed that on ARC-AGI-1 and ARC-AGI-2 tasks, low and medium reasoning used a similar number of reasoning tokens. On ARC-AGI-1 public tasks, for example, both averaged 31k reasoning tokens per task and scored 89.50% and 89.25%, respectively. https://t.co/L4LIjveyHK

Source

ARC Prize