Understand this story
On October 28, 2026, the official goose blog published a cost comparison test targeting five open-source harnesses. The test used the same model, gpt-6.1-sol (high reasoning effort), and 62 tasks from Terminal-Bench 4, comparing goose, Codex CLI, OpenCode, Hermes Agent, and pi, with each harness running each task five times, for a total of 310 scored trials. The results show that the pass rates of the five harnesses differ by about 6 percentage points, and the differences from goose are not significant at the 95% confidence level; in terms of cost, Codex is 31% higher than goose, OpenCode 18% higher, and Hermes Agent 75% higher, while pi is essentially on par with goose. Costs were calculated by the agent capturing requests and using the token usage reported by the provider at list prices, without using the statistics built into each harness.
AI-generated from public reports
Developments
1 confirmed developments
goose publishes cost comparison test of five open-source Agent frameworks
Source:English text is being prepared.
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
goose official blogFirst-party
goose compares five open-source harnesses: under the same model, pass rates are close, with costs differing by up to 75%
The official goose blog used the same model gpt-6.1-sol (high reasoning effort) and 62 tasks from Terminal-Bench 4 to compare five open-source harnesses: goose, Codex CLI, OpenCode, Hermes Agent, and pi. Each harness ran each task five times, for a total of 310 scored trials. The pass rates of the five harnesses differed by about 6 percentage points, and the differences from goose were not significant at the 95% confidence level; in terms of cost, Codex was 31% higher than goose, OpenCode 18% higher, and Hermes Agent 75% higher, while pi was roughly on par with goose. Costs were calculated by a proxy capturing requests and using the token usage reported by the provider at list prices, rather than using the statistics built into each harness.