Understand this story
On October 28, 2026, Nathan Lambert shared his views on whether private benchmarks or open-source benchmarks are better. He noted that private benchmarks often show open-source models lagging further behind, but the problem with private benchmarks is that there is no community to help find and fix the inevitable bugs. Lambert said he is 50/50 on whether private benchmarks or open-source benchmarks are the better approach. Currently, there has been no further evidence-based development on this topic.
AI-generated from public reports
Latest development · Report
Nathan Lambert: Private benchmarks and open-source benchmarks each have their trade-offsDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
Nathan Lambert
Nathan Lambert: Private benchmarks and open-source benchmarks each have their trade-offs
Nathan Lambert pointed out that private benchmarks often show open-source models lagging further behind, but the problem with private benchmarks is that there is no community to help find and fix the inevitable bugs. He said he holds a 50/50 view on whether private benchmarks or open-source benchmarks are the better approach.