Understand this story
Ryan Hanrui Wang, Senior Vice President of Applied AI at Nebius, proposed that evaluating open models should combine industry benchmarks, internal benchmarks, and testing frameworks. He noted that public benchmarks only reflect a model's best-case performance and cannot show whether agents can accurately complete tasks in production environments, so the entire inference stack should be evaluated. He also suggested using large models as references to scan multiple models and find the scale sweet spot; for example, 10B parameters may already be sufficient for customer service tasks, while 100B or even 1T parameters bring only marginal accuracy gains. The related report also mentioned that Nebius acquired inference optimization startup Eigen AI for approximately $643 million and provides inference services through the Token Factory platform.
AI-generated from public reports
Latest development · Report
Evaluate the entire inference stack, not just open-source model benchmarksDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 21
The New Stack
Evaluate the entire inference stack, not just open-source model benchmarks
Ryan Hanrui Wang, Senior Vice President of Applied AI at Nebius, points out that public benchmarks only reflect a model's best-case performance and cannot show whether agents can accurately complete tasks in production environments; therefore, the entire inference stack should be evaluated using internal benchmarks and harnesses. Nebius acquired inference optimization startup Eigen AI for approximately $643 million and provides inference services through its Token Factory platform. He recommends using a large model as a reference to scan multiple models and find the scale sweet spot; for example, 10B parameters may already be sufficient for customer service tasks, while 100B or even 1T parameters bring only marginal accuracy gains.