AI News

Trending story

Together AI open-sources its agent evaluation build package

1 reports1 story updatesUpdated Nov 16 08:00

Understand this story

AI overview

On November 16, 2026, Together AI publicly released its agent evaluation methodology and announced plans to open-source its evaluation toolkit next week. According to the Together AI Blog, the methodology builds prompt lists for common development tasks, runs each of three models twice under three configurations—baseline, documentation MCP, and skills package—and uses an LLM judge to determine pass, incomplete, or fail. The article reviews two cases: in a GPU cluster pricing task, the skill file pointed to a marketing page rather than the price list; after adding two sentences of guidance, all models passed at 100%; API error messages led the agent to a nonexistent route, which has been fixed. After 14 skills were merged into a unified skills package, the first version's pass rate dropped to 75% and the per-run cost doubled; after compressing routes, updating example models, and fixing example code, the pass rate rose to 97.2% and the cost returned to its original level.

AI-generated from public reports

Latest development · Report

How Together AI Does Agent Evaluation: Small Changes Raise Task Pass Rate from 75% to 97%

Developments

1 confirmed developments

  1. Together AI open-sources its agent evaluation building package

    Source:Together AI Blog

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 16

  1. Together AI BlogFirst-party

    How Together AI Does Agent Evaluation: Small Changes Raise Task Pass Rate from 75% to 97%

    Together AI has disclosed its agent evaluation method: it builds a prompt list for common development tasks, runs each of three models twice under three configurations—baseline, documentation MCP, and skills package—and uses an LLM judge to determine pass, incomplete, or fail. The article reviews two cases: first, in a GPU cluster pricing task, the skill file pointed to a marketing page rather than the price list; after adding two sentences of guidance, all models passed 100%; second, an API error message led the agent to a nonexistent route, which has been fixed. After merging 14 skills into one unified skills package, the first version's pass rate dropped to 75% and the per-run cost doubled; after compressing routes, updating the example model, and fixing the example code, the pass rate rose to 97.2% and the cost returned to its original level. The author says that on September 30 the three models' scores differed by only a few percentage points, and that after adding documentation and skill guidance the pass rate rose from 82% to 96% and 97%, and that they plan to open-source their evaluation toolkit next week.

Related stories

  1. Related storiesAnthropic releases Claude Opus 5.5
  2. Related storiesLangChain's OpenWiki generates a Markdown knowledge base from code repositories
  3. Related storiesEnglish translation is being prepared.
  4. Related storiesQwen Releases Qwen4-Max Preview Version