Understand this story
On November 16, 2026, Together AI publicly released its agent evaluation methodology and announced plans to open-source its evaluation toolkit next week. According to the Together AI Blog, the methodology builds prompt lists for common development tasks, runs each of three models twice under three configurations—baseline, documentation MCP, and skills package—and uses an LLM judge to determine pass, incomplete, or fail. The article reviews two cases: in a GPU cluster pricing task, the skill file pointed to a marketing page rather than the price list; after adding two sentences of guidance, all models passed at 100%; API error messages led the agent to a nonexistent route, which has been fixed. After 14 skills were merged into a unified skills package, the first version's pass rate dropped to 75% and the per-run cost doubled; after compressing routes, updating example models, and fixing example code, the pass rate rose to 97.2% and the cost returned to its original level.
AI-generated from public reports
Latest development · Report
How Together AI Does Agent Evaluation: Small Changes Raise Task Pass Rate from 75% to 97%Developments
1 confirmed developments
Together AI open-sources its agent evaluation building package
Source:Together AI Blog
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 16
Together AI BlogFirst-party
How Together AI Does Agent Evaluation: Small Changes Raise Task Pass Rate from 75% to 97%
Together AI has disclosed its agent evaluation method: it builds a prompt list for common development tasks, runs each of three models twice under three configurations—baseline, documentation MCP, and skills package—and uses an LLM judge to determine pass, incomplete, or fail. The article reviews two cases: first, in a GPU cluster pricing task, the skill file pointed to a marketing page rather than the price list; after adding two sentences of guidance, all models passed 100%; second, an API error message led the agent to a nonexistent route, which has been fixed. After merging 14 skills into one unified skills package, the first version's pass rate dropped to 75% and the per-run cost doubled; after compressing routes, updating the example model, and fixing the example code, the pass rate rose to 97.2% and the cost returned to its original level. The author says that on September 30 the three models' scores differed by only a few percentage points, and that after adding documentation and skill guidance the pass rate rose from 82% to 96% and 97%, and that they plan to open-source their evaluation toolkit next week.