TypeSafe AI: How to Judge Whether a Benchmark Is Good or Bad
TypeSafe AI responds to Tim Neumann's discussion about its CEO not trusting public benchmarks, proposing that a good benchmark should have three qualities: timeliness (reflecting only a snapshot of capability at a given moment), groundedness (directly measuring the tasks and data you care about), and specificity (targeting the desired outcome rather than its proxy metrics).
View original postTypesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 https://t.co/Qd2g0Hwp4V
View original
Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 https://t.co/Qd2g0Hwp4V
Body
who benchmarks the benchmarkers?
a good eval is:
- ephemeral. it snapshots a moment in time and capabilities. - grounded. it directly measures performance on tasks and data you care about. - specific. it operates on the desired results and not some proxy of them.
take notes!