Understand this story
ARC Prize encountered API issues while testing Gemini 4 Hydrogen, preventing the ARC-AGI-3 test from being completed. The disclosed issues included the API frequently returning multi-action function calls, MALFORMED_FUNCTION_CALL errors, and repeated actions triggering a RECITATION finish reason. In the completed evaluations, Gemini 4 Hydrogen scored 92.5% on ARC-AGI-2 at a cost of $1.26/task and 98.5% on ARC-AGI-1 at a cost of $0.18/task. The input contains no facts about subsequent developments.
AI-generated from public reports
Latest development · Report
ARC Prize Evaluates Gemini 4 Hydrogen, Revealing ARC-AGI Scores and API ErrorsDevelopments
1 confirmed developments
ARC Prize Encountered API Issues While Testing Gemini 4 Hydrogen
Source:ARC Prize
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 21
ARC PrizeFirst-partySelected
ARC Prize Evaluates Gemini 4 Hydrogen, Revealing ARC-AGI Scores and API Errors
In ARC-AGI evaluations, ARC Prize measured Gemini 4 Hydrogen at 92.5% on ARC-AGI-2 at a cost of $1.26/task and 98.5% on ARC-AGI-1 at a cost of $0.18/task. Its scores on both tasks at the high reasoning level were no lower than those of Gemini 3.8 Flash, while using approximately 52% fewer reasoning tokens. ARC-AGI-3 testing could not be completed because the API frequently returned multi-action function calls and MALFORMED_FUNCTION_CALL errors, and repeated actions triggered a RECITATION finish reason.
AI Trends