AI Trends

ARC Prize Evaluates Gemini 4 Hydrogen, Revealing ARC-AGI Scores and API Errors

ARC PrizeUpdated Nov 21 01:28

AI summary

In ARC-AGI evaluations, ARC Prize measured Gemini 4 Hydrogen at 92.5% on ARC-AGI-2 at a cost of $1.26/task and 98.5% on ARC-AGI-1 at a cost of $0.18/task. Its scores on both tasks at the high reasoning level were no lower than those of Gemini 3.8 Flash, while using approximately 52% fewer reasoning tokens. ARC-AGI-3 testing could not be completed because the API frequently returned multi-action function calls and MALFORMED_FUNCTION_CALL errors, and repeated actions triggered a RECITATION finish reason.

Why read this

SelectedAI score 66

The evaluation presents Gemini 4 Hydrogen's scores, reasoning token usage, and API errors, enabling comparisons of performance and invocation stability.

Topics

Tags

Body

@GoogleDeepMind We also saw API responses repeat actions and eventually trigger a RECITATION finish reason which is normally associated with potential reproduction of copyrighted material. https://t.co/juTYMPQkYJ

Source

ARC Prize