METR 分析 Claude Code 转录记录，估算 AI 智能体带来的生产力提升上限

Our Work

Risk Assessment; About; Donate; Careers

Human uplift studies like the one we did in 2025 are becoming more expensive as working without AI becomes increasingly costly. In this post, I investigate whether coding agent transcripts could serve as a cheaper alternative for estimating uplift. I prototyped this using 5305 Claude Code transcripts generated in January 2026 by 7 METR technical staff1. I used an LLM judge to estimate how long each task would have taken an experienced software engineer without AI tools, then compared that to the time people actually spent on these tasks to calculate a time savings factor.

Takeaways

This method estimates a time savings factor of ~1.5x to ~13x on Claude Code-assisted tasks for 7 METR technical staff in January 2026 – though this result comes with substantial caveats. I believe the true productivity multiplier is substantially lower, and the time savings factor is a soft upper bound for the true uplift that the individuals experienced; Increased agent concurrency may contribute to a higher time savings factor on the Claude Code-assisted task distributions

Limitations

The time savings factor on the coding agent-assisted task distributions does not equal the productivity multiplier. People likely do not create 10x as much value with AI, even if we observe a 10x time savings factor on tasks that people do with AI. I believe the time savings factor overestimates AI-enabled productivity gains for reasons including:

Task Substitution. With AI assistance, people sometimes complete tasks they wouldn’t do otherwise, such as making nice-to-have tools. These tasks tend to deliver lower counterfactual value, even if the observed time savings factor on these tasks might be high; Task Selection Effects. People don’t spend their entire workday using AI—they use it only on tasks where they expect it to be helpful. Since coding transcripts exclude tasks people do without AI, the time savings factor likely overestimates total AI-enabled uplift; Pre-existing Worker Specialization. While the LLM judge estimates are for “experienced software engineers”, individuals typically do tasks they specialize in, and thus take less time than an experienced SWE on average. Therefore, the measured time savings factor probably overstates their actual time savings; Limited LLM Judge validation. The LLM judge used to estimate human task completion time without AI is validated only on 34 ground truth labels, and may not generalize; Limited data sources. I only analyzed Claude Code transcripts from 7 METR technical staff within a single month. This excluded work done with other coding agents, chat interfaces, and non-coding AI assistance, capturing only a slice of AI-assisted work; Limited ability to identify human-typed messages. I estimate the time people actually spent working with Claude Code based on human-typed messages in transcripts. However, when users engage in advanced workflows, such as spinning up new agent sessions using Claude Code, the model-generated user messages may appear indistinguishable from actual human-typed messages. A spot check of 100 randomly sampled “human-typed” messages from our most advanced user found no obvious misclassifications, so this likely does not significantly affect the results in this post

Methodology

The time savings factor is the ratio between the estimated task time without AI and the estimated task time with AI

My definition of the time savings factor follows Anthropic’s recent work on estimating AI productivity gains from Claude conversations.

I calculated the time savings factor daily for each individual, where:

Time estimate with AI is the amount of time the staff member spent working on tasks with AI assistance that day. I estimate this by summing 10-minute windows that contain at least one user-typed message in the transcript. When a user is active in multiple concurrent sessions during the same window, the window is counted only once, rather than being overcounted across parallel sessions; Time estimate without AI is the hours an experienced software engineer would need to complete the same tasks without AI assistance, estimated by an LLM judge given a summarized transcript

LLM Judge’s estimated time without AI via compressed transcripts

I compressed lengthy transcripts by summarizing assistant actions and outputs with GPT-4o while preserving code diffs. GPT-5 then analyzed the compressed transcript to identify successful and failed subtasks and estimated the time an experienced, high-context engineer would need to produce the net successful output.

To only include the time required to produce the successful outputs, I instructed the judge to exclude the following from time estimates2:

Failed tasks: Work with explicit user disapproval; Agent overhead: Setup tasks like configuring subagents and skills, managing agent sessions, or building multi-agent infrastructure—work that wouldn’t exist without AI; Abandoned work: When the user requests A, then pivots to B, the time for A is excluded; Self-explanation: Summarizing or explaining code written in the same session, since this would not be required if engineers wrote the code themselves; Spurious error correction: Fixing trivial bugs introduced by the agent that an experienced engineer wouldn’t make; Verbose planning: For extensive design documents, estimate the time to reach the design, not to write detailed documentation that the agent produces

See Appendix A for the full prompt.

Validating LLM Judge’s time estimate without AI

Estimating task time is difficult even for humans with full context. I validated the judge against 34 human estimates: four staff members estimated time for 26 of their own transcripts, and one staff member estimated time for 8 transcripts based only on the resulting PRs and issues. These transcripts span feature additions, code exploration, greenfield projects, refactoring, and bug fixes, and some include failed subtasks.

On average, the LLM judge’s predictions fall within 2-3x of human estimates, with occasional larger misses. The r_log is 0.83, higher than the human r_log of 0.67 and Claude 4.5 Sonnet’s r_log of 0.46–0.48 reported by Anthropic on time estimates for software development tasks from JIRA tickets.

However, this validation is limited – 34 samples are small. Additionally, while some human-labeled samples include failed tasks—providing partial signal on the judge’s ability to identify subtask success and failure—I did not explicitly validate this step, which introduces additional uncertainty. View example transcripts, human estimates, and LLM estimates here.

Estimated task minutes also moderately correlate with delivered outputs. I collected the lines of code committed (additions + deletions) per day for one technical staff member in January. The lines of committed code explain ~46% of the variation in estimated task minutes per day. Adding issue counts provides a negligible lift (~1.4%). It’s not surprising that there isn’t a stronger correlation: lines of code vary in difficulty, some code is never committed, and commits don’t always land on the same day the work was done.

Based on these results, I believe the judge produces reasonable estimates for our task distribution. Given the limited judge validation and 2-3x error range, the individual time savings factor should be interpreted as approximate rather than precise. However, they should help us distinguish worlds where individuals experience 1x, 10x, and 100x time savings.

Results

Empirically estimated time savings factor is between ~1.5x and ~13x on these transcripts. This serves as a soft upper bound for both (1) the productivity multiplier on Claude Code-assisted tasks and (2) the individual’s overall AI-enabled productivity uplift.

Bootstrapping over the daily time savings factor for each individual, we observe time savings factors between ~1.5x and ~13x on the tasks contained in the Claude Code transcripts.

For the following reasons, I believe the individual’s time savings factor is a soft upper bound on their productivity uplift for Claude Code-assisted tasks:

As further evidence, I asked several staff members to estimate how long it would have taken them to do the same tasks they did with Claude Code for a few days. See Appendix B for the exact survey question. All self-estimates fell below the bootstrapped point estimate, with most falling below the 90% confidence interval.

People use AI on lower-value tasks. Anthropic’s internal survey reports that 27% of Claude-assisted work wouldn’t have been done otherwise. In our judge validation sample, 3.5 of 8 human-labeled PRs were work the staff member wouldn’t have done without AI, totaling ~47% of the estimated task time without AI for that individual. The methodology overestimates the productivity multiplier on Claude Code-assisted tasks when people use AI to complete low-value but time-consuming work (we call them “Cadillac tasks”); The LLM judge likely overestimates time without AI. In most of the 34 validation samples, the judge predicted longer times than humans estimated. There are several reasons why this might be the case:

The LLM judge tends to assume a task is successful even if the user quits an agent session without explicit disproval; Staff tend to specialize in the tasks they do, completing them faster than a generic “experienced software engineer.”; The judge evaluates individual transcripts in isolation, so it misses cases when progress is undone between sessions or when users start a new session as a retry of a previously failed session; Despite excluding explicit agent setup tasks, some agent-induced overhead is hard to separate—such as breaking work into granular steps a human would skip otherwise

The additional reason below supports that an individual’s time savings factor is a soft upper bound on their productivity uplift across all work:

Most staff don’t spend their entire day coding with Claude Code. For example, suppose a researcher spends 4 hours coding with Claude Code and 4 hours in meetings. Then, a 4x time savings factor on Claude Code tasks and 1x on everything else only yields 2.5x total uplift. However, this logic cuts both ways: if the same person also spends 2 hours on AI-assisted non-coding tasks (e.g., via chat interfaces) and experiences much higher uplift there, their total productivity gain could exceed the coding-only time savings factor. (Though based on anecdotal evidence, this is unlikely to be true for the staff participating in this study.)

Higher agent concurrency is associated with a higher time savings factor

Users can interact with Claude Code through one or more main agents, which can launch subagents in parallel to complete subtasks. I define concurrency as the number of active agents per person, reporting both the main agent concurrency and the total concurrency (main plus subagents). Daily average concurrency is computed as a time-weighted mean over periods when at least one session is active. Across the 7 staff members, a higher daily average concurrency (main and total) is associated with a higher time savings factor. Each point in the graph below represents one individual’s average concurrency on a given day.

Technical Staff A, who has the highest time savings factor, averages at least 2.32 main agents and 2.74 total agents on active days. The remaining staff average between 1.05 and 1.52 main agents and 1.07–1.57 total agents. See Appendix C for each individual’s average concurrency and Appendix D for qualitative descriptions of technical staff A’s workflow.

While causality can’t be established, I believe workflows optimized for agent concurrency may substantially increase the time savings factor for our technical staff. Though, as before, it is unclear whether this represents a genuine productivity improvement, because additional tasks completed via parallelism may be lower-value.

Discussion

Uplift studies may underestimate uplift by not accounting for concurrency

METR’s human uplift RCT randomized developers to complete issues one at a time, with or without AI. The correlation between concurrency and the time savings factor suggests this design may understate AI’s potential impact. Future uplift studies we design should allow participants to work on multiple issues simultaneously, or assign larger issues amenable to concurrent agent use. Similarly, transcript-based analyses should account for parallelism rather than treating transcripts from the same individual as independent observations.

Measuring AI capabilities on real-world tasks via transcripts

Autonomous capabilities benchmarks have many limitations: they saturate quickly as models improve, they require significant upfront effort to create, and it’s hard to translate benchmark performance to real-world capabilities.

Real-world transcripts may be a complementary source of evidence in understanding autonomous capabilities. Relative to benchmarks, transcripts have several advantages:

They’re generated naturally as a byproduct; They reflect the real-world, messy task distribution that we care about; They can provide information on how people adopt AI in their workflows

However, transcripts also have important limitations. Most significantly, people mostly use AI on tasks where they expect it to succeed. Observing high success rates on tasks in transcripts doesn’t tell us how well AI does on the tasks people don’t attempt with AI.

Despite the limitations, I believe transcripts can be useful for understanding autonomous capabilities. Impressive one-off demonstrations still provide genuine evidence of capability. Transcripts can also capture failure modes that curated benchmarks may miss.

To build on top of this preliminary research, one could:

Expand the analysis to serve not only Claude Code transcripts but also other command-line and IDE coding agent transcripts; Creating a better LLM judge and conducting further validation. The current judge has many known limitations, and should be validated on a larger number of human time estimates on a broader task distribution; Validate the methodology against an RCT uplift study. How well does the estimated time savings factor compare to the actual uplift ratio? Can we predict the uplift ratio based on the estimated time savings factor?; Better understand the effect of task substitution. One might survey developers directly and ask how much more valuable output they deliver as a result of having access to AI tools; Analyze what kinds of tasks people use AI agents for, and how often agents succeed at different task types. This can help us better understand the trend of autonomous capabilities

I would be excited for AI developers to conduct similar experiments on internal transcripts and inform the world about their speedup.

Conclusion

From this exploratory analysis on METR’s internal transcript data, we observe that people are substantially sped up on the tasks they use Claude Code for. The actual productivity uplift is likely much lower (but likely still positive) due to factors such as task substitution, task selection, and worker specialization. I believe that the empirically measured time savings factor for an individual serves as a soft upper bound for both their productivity multiplier on Claude Code-assisted tasks and across all work. Further research on coding transcripts would be valuable, and we would be excited to see AI developers publish empirical speedup measurements from their own transcripts.

Appendix A: LLM Judge prompts

This prompt is used to summarize assistant turns between human-typed user turns:

This prompt is used for estimating time without AI:

Appendix B: Staff daily time savings factor self-estimates survey

This is the exact question I asked the technical staff at the end of their workday or the start of the following workday:

Appendix C: Individual agent concurrency statistics

Average agent concurrency numbers on a typical active day for each individual, alongside their time savings ratio, active days, and average number of active hours per day.

Qualitatively, Technical Staff A optimizes their workflow for concurrency in the following ways:

He context-switches frequently between projects and uses git worktrees to work on multiple PRs in the same repo; He front-loads effort on detailed plans and verification instructions, enabling agents to sometimes run autonomously for 1–3+ hours while he spins up additional sessions; He often has 10+ terminal sessions open at the same time to monitor multiple agents simultaneously; He employs techniques like “Ralph Wiggum,” which repeatedly feeds an agent a prompt file via a while true loop, allowing iterative improvement until completion. We note that this technique may artificially inflate total concurrency when the agent keeps running without doing real work

See Appendix E for visualizations of Technical Staff A’s per-transcript uplift distribution and daily estimated minutes and time savings factor.

Appendix E: Detailed results for technical staff A

I briefly investigated whether Technical Staff A’s high time savings factor is driven by extreme outliers. To calculate per-transcript time savings, I computed concurrency-adjusted human-active minutes for each session. For each 10-minute window, I counted the number of transcripts the user actively interacted with (n), and attributed 10/n minutes to each transcript. This filtering excluded sessions with no active human participation, such as those running in Ralph Wiggum loops. I also excluded sessions where the LLM judge estimated 100x time savings factors. The highest (196x) involved just 2 human messages and was estimated at 785 minutes without AI—the user asked the agent to evaluate 23 knowledge graph tools against an evaluation spec. Other high-uplift transcripts involved large implementations or refactorings based on pre-written plans.

The graph below shows the variations in the daily time savings ratio for the same technical staff.

This excludes transcripts containing confidential information. ↩; Some agent workflows involve idle loop iterations where the agent repeatedly confirms completion. These don’t contribute substantially to the total time estimates and are not explicitly excluded in this version of the LLM judge. ↩

Want to contribute to this work? METR is hiring: View open roles

Featured research

METR researches, develops and runs cutting-edge tests of AI capabilities, including broad autonomous capabilities and the ability of AI systems to conduct AI R&D.

A survey of 349 technical workers finds a median 1.4–2x self-reported change in value of work due to AI tools, expected to grow over time, though there are reasons to be skeptical of the magnitude.

Early Work on Monitorability Evaluations

We show preliminary results on a prototype evaluation that tests monitors' ability to catch AI agents doing side tasks, and AI agents' ability to bypass this monitoring.

How Does Time Horizon Vary Across Domains?

We build on our time-horizon work and analyze 9 benchmarks for scientific reasoning, math, robotics, computer use, and self-driving in terms of time-horizon trends; we observe generally similar rates of improvement to the 7-month doubling time in our original time-horizon work.