Anthropic proposes the Agent4Sci evaluation framework: AI scientific research improvements mostly come from engineering tricks

Frontier coding agents are turning machine learning research into automatically executable workflows, but existing research mostly focuses on well-structured codebases, single tasks, or the computer science domain. For research projects scattered across different disciplines and lacking well-established training and evaluation pipelines, whether agents can genuinely improve models still lacks systematic measurement. Testing agents directly on raw research codebases is also unreliable: when the original repositories lack clear metric definitions or standardized data splits, agents may exploit evaluation loopholes to achieve superficial gains rather than genuinely improving the model.

To address this, the Anthropic team proposed Agent4Sci, a long-horizon automated hill-climbing evaluation framework for scientific machine learning (ML for Science) tasks, covering 11 scientific domains, 20 benchmarks, and 25 tasks.

In the experiments, four frontier agents collectively improved all 20 benchmarks, with Claude Opus achieving an average relative improvement of 10.4%. However, by analyzing the agents' run trajectories, the research team found that the improvements mainly came from routine machine learning engineering techniques rather than research-level insights. After adding three plugin skills, Opus's average relative improvement further rose to 20.1%. This work was accepted to the NeurIPS 2026 Evaluations and Datasets track.

The construction of Agent4Sci proceeds in two steps: first standardizing the research tasks, then using a three-stage protocol to isolate hill-climbing, evaluation, and auditing, limiting the room for cheating exposed in pilot experiments.

The first step is four-stage screening plus manual standardization. The research team screened benchmarks through four stages: domain classification, automated screening, manual verification, and in-depth review. Taking the NeurIPS 2025 Datasets and Benchmarks track as an example, the research team ultimately selected 10 benchmarks from 497 candidate papers, a rejection rate of about 98%. Every selected task must pass reproduction of the paper's baseline and complete standardization of data splits, runtime environment, and training pipeline.

The second step is the three-stage evaluation protocol. First, the hill-climbing agent receives the paper, code, training data, and validation data, and within a fixed resource budget modifies the code, trains models, and submits results, but cannot access the test set. Then, an independent evaluation agent uses a standardized evaluation pipeline to compute final metrics on the held-out test set. Finally, an auditing agent inspects run logs and submitted artifacts to identify evaluation gaming and other academic misconduct.

Figure | The Agent4Sci evaluation protocol, with the hill-climbing, evaluation, and auditing agents isolated from one another

This design specifically addresses the evaluation loopholes exposed in pilot experiments. The research team further used auditing agents from three model families—Claude, Codex, and Gemini—to cross-review some run trajectories; the audit conclusions remained consistent, and no runs were flagged. However, independent evaluation and auditing do not mean all covert gaming behavior can be ruled out.

The experiments compared Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro, and Codex (GPT-5.4). All four ran through their respective official CLIs at maximum reasoning effort. Each experiment provided 8 H200 GPUs and a 12-hour budget, with no human involvement in specific decisions during the experiment. The four agents' average relative improvements were 10.4%, 9.6%, 6.8%, and 4.4%, respectively. Not every agent improved on every benchmark, but all 20 benchmarks were improved by at least one agent, with individual improvements ranging from 1.1% to 64.4%.

Figure | Average relative improvement of four frontier agents on Agent4Sci

Specific cases show that improvements can come from routine model optimization or from more targeted task modifications. In the PFDelta power flow estimation task, Opus reduced the average power balance loss by 64.4% by adjusting auxiliary losses, increasing the weight of the mean squared error loss, and switching to cosine annealing with warmup. In the cone detection task of Mars-Bench, Opus replaced the original RetinaNet with Faster R-CNN, which is better suited to the task, raising mAP from 16.5 to 30.4.

However, trajectory analysis revealed the limitations behind these gains. Agents heavily adopted mature methods such as learning rate tuning, deepening and widening models, backbone replacement, and weight decay, while interventions targeting specific scientific problems were relatively rare. There were also clear differences in how different agents worked: Opus continuously monitored logs and iterated on experiments, Codex tended to submit long-running scripts earlier and then wait for execution to finish, and Gemini fell somewhere in between.

In response to problems that recurred in the trajectories, the research team designed three lightweight plugin skills, for research ideation, experiment management, and scheduled checklists, respectively. The plugin skills nearly doubled the average improvement of two agents. Opus improved on 18 of 20 benchmarks, with PFDelta jumping from 64.4% to 91.0%, MassSpecGym reaching 39.5%, and CarbonGlobe reaching 35.3%. The new approach on PFDelta reads more like physical reasoning: the agent identified that the power balance loss can be derived from Kirchhoff's laws and requires no labels, refactored it as the primary training loss, used GNN predictions at inference time to provide a warm start for the AC power flow equations, and progressively refined the voltage variables to drive the physical residual toward zero.

Figure | Comparison of improvement magnitudes per benchmark between default runs and runs with plugin skills added

The research team stated that although the performance gains mainly come from machine learning engineering rather than scientific reasoning, Agent4Sci still turns the ability to improve models in automated scientific research into a reproducible, auditable evaluation problem. It not only measures whether agents can raise metrics, but also attempts to distinguish whether improvements come from mature engineering techniques or from understanding of the scientific problem itself.

In the future, whether coding agents can move from executing experiments more efficiently to proposing more valuable research ideas will become an important question for measuring automated scientific research capability.

For more technical details, see the original paper.

This article is from the WeChat public account "Academic Headlines" (ID: SciTouTiao), author: Zhang Jiayi, published by 36Kr with authorization.