Nathan Lambert: Private benchmarks and open-source benchmarks each have their trade-offs
AI summary
Nathan Lambert pointed out that private benchmarks often show open-source models lagging further behind, but the problem with private benchmarks is that there is no community to help find and fix the inevitable bugs. He said he holds a 50/50 view on whether private benchmarks or open-source benchmarks are the better approach.
Body
Private benchmarks often show open models are further behind, but the problem with private benchmarks is you don’t have the community to squash the inevitable bugs for you. I'm 50/50 on if I think private or open-source is the best approach. https://t.co/DbvE6Tk5qL