AI News

Thomas Wolf 发布《The Ultimate MoE Training Handbook》开源训练手册

Thomas Wolf更新于 11月6日 04:25

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

AI 导读

Thomas Wolf 发布免费开源的《The Ultimate MoE Training Handbook》,基于 80 万 GPU 小时实验整理而成,围绕训练团队常遇到的问题组织内容,包括专家数量与规模、all-to-all 通信开销、路由函数选择、辅助损失与 DeepSeek 的无损偏置、稠密 checkpoint 升级以及推理时减少每 token 专家数等。手册约 7 万字,含 55 个交互图表和可运行代码,并附有作者自己实验跑出的数据。

推荐理由

AI 评分 62

作者用 80 万 GPU 小时实验整理出 MoE 训练手册,覆盖专家配置、通信开销与路由选择等团队常遇问题。

主题

标签

正文

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://t.co/6XihlUrj92

来源

Thomas Wolf