Rohan Paul on Sam Altman's Remarks About Local Inference
Rohan Paul quotes Sam Altman's view on local inference: from a global resource allocation perspective, having the world's limited DRAM sitting idle in everyone's computers and only 5% utilized is terrible. Rohan Paul adds that on a per-token cost basis, data centers are clearly superior, and a local model with batch size 1 is wasteful by any utilization metric, but it is also the only part of the entire tech stack that no one can take down on you.
Body
Sam Altman on local inference.
"from a global allocation perspective, it's quite terrible to have everybody, a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time"
per token, the datacenter wins easily. A local model at batch size 1 is wasteful by any utilization metric, and it’s also the only part of the stack nobody can deprecate on you. https://t.co/w9A93jliU9