TypeSafe AI：如何评判基准测试的好坏
who benchmarks the benchmarkers?
TypeSafe AI 回应 Tim Neumann 关于其 CEO 不信任公开基准测试的讨论，提出好评测应具备三点：时效性（只反映某一时刻的能力快照）、有据可依（直接测量你关心的任务与数据）、具体（针对期望结果而非其代理指标）。
查看原帖TypeSafe 的 CEO @CompleteSkeptic 不信任公开基准测试。 Diogo，我也不信。 于是 @p0 和我写了一篇博客，基于我们亲自审核并反复迭代过的真实公司训练数据集，评估了 8 个决策模型。 不过话说回来，Jev 依然稳居榜首 👑 https://t.co/Qd2g0Hwp4V
查看原文
Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 https://t.co/Qd2g0Hwp4V
正文
who benchmarks the benchmarkers?
a good eval is:
- ephemeral. it snapshots a moment in time and capabilities. - grounded. it directly measures performance on tasks and data you care about. - specific. it operates on the desired results and not some proxy of them.
take notes!