AI Trends

ARC Prize Was Unable to Complete Gemini 4 Hydrogen's ARC-AGI-3 Test Due to API Issues

ARC PrizeUpdated Nov 21 01:28

AI summary

ARC Prize encountered API issues while testing Gemini 4 Hydrogen and was unable to complete the ARC-AGI-3 test. ARC-AGI-3 requires responses to be strings containing only a single action, but the model frequently returned function calls containing multiple actions. The API also repeatedly returned MALFORMED_FUNCTION_CALL, along with repeated actions and a RECITATION finish reason.

Why read this

AI score 51

The testing exposed incompatibilities between the model's output format and the evaluation interface requirements, providing a concrete case study for reproducing results and designing agent evaluation workflows.

Topics

Tags

Body

@GoogleDeepMind During ARC-AGI-3 testing we encountered API issues that prevented us from completing testing.

For example, ARC-AGI-3 expects responses to be strings containing a single action, but Gemini 4 Hydrogen frequently returned function calls containing multiple actions. https://t.co/8sHLkzIaNa

Source

ARC Prize