Rohan Paul says curating test datasets from production usage is very practical

Oct 28

1. 01:13

   Rohan Paul

   Rohan Paul comments on Judgment's organization of test sets from production data

   Judgment Labs announced that it is opening to the public, describing itself as a monitoring and improvement layer for long-horizon agents that can extract information such as user requirements and failure scenarios from agent traces. Rohan Paul commented that being able to more conveniently organize test datasets from production usage is quite practical, because users will raise all kinds of requirements that no one thought to write down in advance, and incorporating these into testing would be better.

   Expand summary
2. 00:50

   Testing Catalog

   Judgment Labs is now generally available, providing a continuous improvement toolchain for AI agents

   Judgment Labs has officially opened for general access (GA), positioned as a continuous improvement toolchain for AI agents; agent clusters can traverse production traces to discover failures, classify and investigate them, and test fixes. Its Judgment MCP can be used with Claude, Codex, Cursor, or any MCP client. Publisher Alex Shan said that long-running agent systems generate large amounts of trace data, which records user requirements, when agents go wrong, and how agents interact with the external world.

   Expand summary