Rohan Paul comments on Judgment's organization of test sets from production data
Judgment Labs announced that it is opening to the public, describing itself as a monitoring and improvement layer for long-horizon agents that can extract information such as user requirements and failure scenarios from agent traces. Rohan Paul commented that being able to more conveniently organize test datasets from production usage is quite practical, because users will raise all kinds of requirements that no one thought to write down in advance, and incorporating these into testing would be better.
View original postToday, @JudgmentLabs opens for general access. We believe long running agent systems will generate more exhaust data than the world can currently process. Hidden within agent traces are rich records of how to make these systems better: what users ask for, when agents stumble, and how agents interact with the external world. Judgment is the most performant monitoring and improvement layer for long-horizon agents. We’re excited to see what you build!
View original
Today, @JudgmentLabs opens for general access. We believe long running agent systems will generate more exhaust data than the world can currently process. Hidden within agent traces are rich records of how to make these systems better: what users ask for, when agents stumble, and how agents interact with the external world. Judgment is the most performant monitoring and improvement layer for long-horizon agents. We’re excited to see what you build!
Body
Making it easier to curate test datasets from production usage sounds pretty handy.
People ask for all sorts of things nobody thought to write down in advance. Would be nice to get more of those into the tests