Understand this story
Michael Pfaffenberger of Pydantic built a benchmark comparing code mode with direct tool calling, using 5 small Pydantic AI applications and 8 models, completing a total of 5,276 scored runs. The results show that on long tasks, code mode reduced average cost by 58%, cut time by 51%, and improved the success rate by 16 percentage points; however, on short tasks with 5 items, 3 of the 5 tasks were instead 21% to 38% more expensive. Michael Pfaffenberger believes that models are better at writing loops than at acting as the loop, and that the gap widens as tasks get longer. The benchmark results have now been published in an official Pydantic AI article.
AI-generated from public reports
Latest development · Report
Pydantic compares code mode and tool calling across 5,276 runsDevelopments
1 confirmed developments
Michael Pfaffenberger builds a benchmark for code mode and tool calling
Source:English text is being prepared.
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Official Pydantic AI articleFirst-partySelected
Pydantic compares code mode and tool calling across 5,276 runs
Michael Pfaffenberger of Pydantic used 5 small Pydantic AI applications and 8 models to conduct 5,276 scored runs, comparing code mode with direct tool calling. On long tasks, code mode reduced average cost by 58%, latency by 51%, and increased success rate by 16 percentage points; however, for short tasks with 5 items, 3 out of 5 tasks were instead 21% to 38% more expensive. The author argues that models are better at writing loops than at acting as the loop, and the gap widens as tasks get longer.