Rohan Paul 评 Sam Altman 谈本地推理
Sam Altman on local inference.
AI 导读
Rohan Paul 引述 Sam Altman 对本地推理的看法：从全球资源分配角度看，让全世界有限的 DRAM 都躺在各人电脑里、只用到 5% 是很糟糕的。Rohan Paul 补充说，按每 token 成本算数据中心明显占优，batch size 1 的本地模型在任何利用率指标上都是浪费，但它也是整个技术栈里唯一没人能对你下架的部分。
正文
Sam Altman on local inference.
"from a global allocation perspective, it's quite terrible to have everybody, a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time"
per token, the datacenter wins easily. A local model at batch size 1 is wasteful by any utilization metric, and it’s also the only part of the stack nobody can deprecate on you. https://t.co/w9A93jliU9