ARC Prize publishes evaluation results for DeepSeek V4.1 Flash 1017

Nov 3

1. 05:31

   TeortaxesRelated discussion

   Teortaxes joined this discussion ↗

   This post contributes to heat; it is not a formal report or confirmed fact.

2. 04:57

   ARC PrizeFirst-partySelected

   ARC Prize announces evaluation results of DeepSeek V4.1 Flash 1017 on ARC-AGI

   ARC Prize announces the verified results of DeepSeek V4.1 Flash 1017 on ARC-AGI: ARC-AGI-2 at 78.1% with $0.16 per task, ARC-AGI-1 at 94.5% with $0.06 per task, and ARC-AGI-3 at 5.8% with $1.2K under the provider adapter harness and 0.2% with $1.0K under the standard harness. The model leads open-weight models on ARC-AGI-3 and ARC-AGI-2, and matches the best score of open-weight models on ARC-AGI-1. Compared with the original V4.1 Flash from September 10, 2026, 1017's ARC-AGI-3 semi-private set score rose from 2.68% to 5.81%, while its public set score fell from 11.05% to 8.97%; at maximum reasoning effort, the reasoning tokens consumed by its first action decreased by an average of 36% across 25 public games and by 48.6% across 55 semi-private games.

   Expand summary
3. 04:57

   ARC PrizeFirst-party

   ARC Prize announces evaluation results of DeepSeek V4.1 Flash 1017 on ARC-AGI

   ARC Prize announces the evaluation results of DeepSeek V4.1 Flash 1017 on ARC-AGI: it scores 5.81% on the ARC-AGI-3 semi-private set, 78.1% on ARC-AGI-2, and 94.5% on ARC-AGI-1, leading among open-weight models. Compared with the original V4.1 Flash released on September 10, 2026, its ARC-AGI-3 semi-private set score rose from 2.68% to 5.81%, while its public set score fell from 11.05% to 8.97%. At maximum reasoning effort, the model's reasoning tokens consumed for the first action decreased by an average of 36% across 25 public games and by 48.6% across 55 semi-private games.

   Expand summary
4. 04:57

   ARC PrizeFirst-party

   DeepSeek V4.1 Flash 1017 Doubles Its ARC-AGI-3 Score

   ARC Prize's comparison shows that DeepSeek V4.1 Flash 1017's ARC-AGI-3 semi-private set score rose from 2.68% to 5.81%, while its public set score fell from 11.05% to 8.97%. Both tests used the provider adapter harness, max reasoning, and Baseten pricing ($0.30/M input, $1.20/M output).

   Expand summary
5. 04:57

   ARC PrizeFirst-party

   ARC Prize announces evaluation results of DeepSeek V4.1 Flash 1017 on ARC-AGI

   ARC Prize announced DeepSeek V4.1 Flash 1017's results on the ARC-AGI series of benchmarks: 5.81% on ARC-AGI-3, 78.1% on ARC-AGI-2, and 94.5% on ARC-AGI-1, leading among open-weight models. Compared with the original DeepSeek V4.1 Flash released on September 10, 2026, 1017's ARC-AGI-3 semi-private set score rose from 2.68% to 5.81%, while its public set score fell from 11.05% to 8.97%. At maximum reasoning effort, the reasoning tokens used by 1017 for its first action decreased by an average of 36% across 25 public games and by 48.6% across 55 semi-private games.

   Expand summary
6. 04:57

   ARC PrizeFirst-party

   ARC Prize announces evaluation results of DeepSeek V4.1 Flash 1017 on ARC-AGI

   ARC Prize announces the validation results of DeepSeek V4.1 Flash 1017 on ARC-AGI: it scores 5.8% on ARC-AGI-3 at a cost of $1.2K (provider adapter harness), and 0.2% at $1.0K under the standard harness. The model leads open-weight models on ARC-AGI-3 with 5.81%, scores 78.1% on ARC-AGI-2, and ties for the highest open-weight score on ARC-AGI-1 with 94.5%. Compared with the original V4.1 Flash from September 10, 2026, 1017's ARC-AGI-3 semi-private set score rose from 2.68% to 5.81%, while its public set score fell from 11.05% to 8.97%.

   Expand summary