AI Trends

Trending story

ARC Prize Encounters API Issues While Testing Gemini 4 Hydrogen

1 reports1 story updatesUpdated Nov 21 00:57

Understand this story

AI overview

ARC Prize encountered API issues while testing Gemini 4 Hydrogen, preventing the ARC-AGI-3 test from being completed. The disclosed issues included the API frequently returning multi-action function calls, MALFORMED_FUNCTION_CALL errors, and repeated actions triggering a RECITATION finish reason. In the completed evaluations, Gemini 4 Hydrogen scored 92.5% on ARC-AGI-2 at a cost of $1.26/task and 98.5% on ARC-AGI-1 at a cost of $0.18/task. The input contains no facts about subsequent developments.

AI-generated from public reports

Latest development · Report

ARC Prize Evaluates Gemini 4 Hydrogen, Revealing ARC-AGI Scores and API Errors

Developments

1 confirmed developments

  1. ARC Prize Encountered API Issues While Testing Gemini 4 Hydrogen

    Source:ARC Prize

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 21

  1. ARC PrizeFirst-partySelected

    ARC Prize Evaluates Gemini 4 Hydrogen, Revealing ARC-AGI Scores and API Errors

    In ARC-AGI evaluations, ARC Prize measured Gemini 4 Hydrogen at 92.5% on ARC-AGI-2 at a cost of $1.26/task and 98.5% on ARC-AGI-1 at a cost of $0.18/task. Its scores on both tasks at the high reasoning level were no lower than those of Gemini 3.8 Flash, while using approximately 52% fewer reasoning tokens. ARC-AGI-3 testing could not be completed because the API frequently returned multi-action function calls and MALFORMED_FUNCTION_CALL errors, and repeated actions triggered a RECITATION finish reason.

Related stories

  1. Related storiesGemini 4 Hydrogen Matches or Surpasses Its Predecessor’s Scores on Two ARC-AGI Benchmarks
  2. Related storiesGemini 4 Neon Begins Rolling Out on the Gemini and Antigravity Platforms
  3. Related storiesAnthropic releases Claude Opus 5.5
  4. Related storiesLogan Kilpatrick announced the launch of Gemini Decision 1 on the Gemini API