Thomas Wolf releases the open-source training handbook "The Ultimate MoE Training Handbook"
Thomas Wolf has released the free, open-source "The Ultimate MoE Training Handbook," compiled from 800,000 GPU hours of experiments and organized around the problems training teams commonly encounter, including the number and size of experts, all-to-all communication overhead, the choice of routing function, auxiliary losses and DeepSeek's loss-free bias, upgrading dense checkpoints, and reducing the number of experts per token at inference time. The handbook is about 70,000 words long, contains 55 interactive charts and runnable code, and includes data from the author's own experimental runs.
Why read this
AI score 62
The author distilled a MoE training handbook from 800,000 GPU hours of experiments, covering common problems teams encounter such as expert configuration, communication overhead, and routing choices.
Topics
Body
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with?
⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
Read it here: https://t.co/6XihlUrj92