Lilian Weng 发布 scaling laws 长文
A super long overdue (3+ years?) post on scaling laws.
AI 导读
一篇拖了很久（3 年多？）的关于 scaling laws 的文章。 算力很贵。scaling laws 是一种帮助我们在投入大规模训练之前，推理数据与模型规模之间最优算力分配的方法。 文章涵盖 scaling laws 预测了什么、算力最优分配如何运作、为什么 Kaplan et al. 与 Chinchilla 结论不一致，以及数据限制 + 拟合细节如何让外推变得棘手。
正文
当前显示原文，中文全文译文尚未就绪。
A super long overdue (3+ years?) post on scaling laws.
Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.
The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky.