AI News

Thomas Wolf releases "The Ultimate MoE Training Handbook"

Ma Dongxi NLPUpdated Nov 6 16:46

English text is being prepared.

AI summary

Thomas Wolf has released the free and open-source "The Ultimate MoE Training Handbook." Based on 800k GPU hours of experiments, it systematically explains how to train Mixture-of-Experts models. The handbook is about 70,000 words long and includes 55 interactive charts and runnable code. It addresses common questions faced by training teams, such as the number and size of experts, all-to-all communication overhead, router choice (softmax, sigmoid, or sqrt(softplus)), aux loss and DeepSeek's loss-free bias, and dense checkpoint upcycle. He points out that 18 of the 20 best open-source models today are MoE.

Why read this

AI score 69

Based on 800k GPU hours of experiments, this handbook answers practical questions in MoE training one by one, including expert configuration, communication overhead, and routing choices.

Topics

Tags

Quoted post

Thomas Wolf

@Thom_Wolf

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://t.co/6XihlUrj92

View original

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://t.co/6XihlUrj92

View original post

Body

裂墙推荐就完事了!

如何训练 Mixture-of-Experts.

The Ultimate MoE Training Handbook

Source

Ma Dongxi NLP