Understand this story
IBM has partnered with Kaggle to establish a continuously updated public leaderboard for IBM's open-source benchmark ScarfBench, with Kaggle independently reproducing and running the benchmark, providing compute resources, and maintaining it over the long term. ScarfBench evaluates the ability of coding agents to migrate enterprise Java applications among Jakarta EE, Quarkus, and Spring, comprising 204 migration tasks and 10 model-and-agent combinations. The leaderboard shows that Gemini 4 Argon with Antigravity ranks first on the smoke metric at 75.2%, followed by Claude Opus 5.5 with Claude Code at 70.5%; on the all-tests pass rate, Claude Opus 5.5 with Claude Code leads at 72.1%. IBM noted that of the 745 failed runs, about 72% were runtime behavior failures, while compilation or packaging failures accounted for about 20%.
AI-generated from public reports
Latest development · Report
IBM and Kaggle Partner to Launch the ScarfBench Continuous Migration LeaderboardDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 27
Unite.AI
IBM and Kaggle Partner to Launch the ScarfBench Continuous Migration Leaderboard
IBM has partnered with Kaggle to establish a continuously updated public leaderboard for IBM's open-source benchmark ScarfBench; Kaggle independently reproduces and runs the benchmark, provides the compute, and maintains it long-term. ScarfBench evaluates the ability of coding agents to migrate enterprise Java applications between Jakarta EE, Quarkus, and Spring, comprising 204 migration tasks and 10 model-agent combinations; Gemini 4 Argon with Antigravity ranks first on the smoke metric at 75.2%, followed by Claude Opus 5.5 with Claude Code at 70.5%, while on the all-tests pass rate Claude Opus 5.5 with Claude Code leads at 72.1%. IBM notes that of the 745 failed runs, about 72% were runtime behavior failures, and compilation or packaging failures accounted for about 20%.