Thomas Wolf releases "The Ultimate MoE Training Handbook"
English text is being prepared.
Thomas Wolf has released the free and open-source "The Ultimate MoE Training Handbook." Based on 800k GPU hours of experiments, it systematically explains how to train Mixture-of-Experts models. The handbook is about 70,000 words long and includes 55 interactive charts and runnable code. It addresses common questions faced by training teams, such as the number and size of experts, all-to-all communication overhead, router choice (softmax, sigmoid, or sqrt(softplus)), aux loss and DeepSeek's loss-free bias, and dense checkpoint upcycle. He points out that 18 of the 20 best open-source models today are MoE.
Why read this
AI score 69
Based on 800k GPU hours of experiments, this handbook answers practical questions in MoE training one by one, including expert configuration, communication overhead, and routing choices.
Topics
View original post18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://t.co/6XihlUrj92
View original
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://t.co/6XihlUrj92
Body
裂墙推荐就完事了!
如何训练 Mixture-of-Experts.
The Ultimate MoE Training Handbook