AI News

Hugging Face releases MoE training handbook, investing about 800,000 GPU hours in experiments

Shao MengUpdated Nov 6 20:55

AI summary

The Hugging Face team invested about 800,000 GPU hours in experiments and compiled the results into a free, open "Ultimate MoE Training Handbook," roughly 70,000 words long, with 55 interactive diagrams, runnable code, and real experimental data. The handbook covers questions such as choosing the number and size of experts, cross-GPU communication slowing down training, routing and load-balancing settings, and whether to convert a dense model or train from scratch. The example in the text uses 64 experts with 8 selected each time and a width of 1024 per expert; the entire example model has about 6.86 billion total parameters and 1.22 billion activated parameters. In one performance analysis, communication not hidden by computation accounted for about half of a step, and a case where communication fell back to ordinary sockets was about 25 times slower.

Why read this

AI score 63

The handbook lays out, item by item with its own experimental data, the trade-offs in MoE training among routing precision, load balancing, and cross-GPU communication, making it easy to compare and troubleshoot.

Topics

Tags

Quoted post

Thomas Wolf

@Thom_Wolf

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://t.co/6XihlUrj92

View original

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: https://t.co/6XihlUrj92

View original post

Body · AI translation

The Hugging Face team invested ~800,000 GPU hours in experiments and compiled the results into a free, openly available "Ultimate MoE Training Handbook."

The full text is about 70,000 words, with 55 interactive diagrams, runnable code, and real experimental data. It is organized around the problems training teams encounter most often: how to choose the number and size of experts, why cross-GPU communication slows training, how to set up routing and load balancing, and whether to convert a dense model or train from scratch. It is an engineering guide that systematically makes scattered MoE training experience public for practitioners to consult repeatedly. https://t.co/CBHSGqSAnY

What MoE actually saves

MoE replaces part of the feed-forward network with multiple experts, and a router selects a few of them for each token, then merges their outputs. As a result, the model can increase its total parameter count without increasing the computation per token by the same proportion.

The example in the article has 64 experts, selects 8 each time, and each expert has a width of 1024. Compared with a dense feed-forward layer of width 8192, it stores about 8 times as many parameters, while the total width of experts used per token is the same. What is saved is activation computation, not the storage requirement for all parameters; the entire example model has about 6.86 billion total parameters and 1.22 billion activated parameters.

The real difficulty lies in three areas

1. Routing must be stable. Expert selection depends on score ranking, and low precision may change close scores and selection results. The handbook therefore emphasizes using FP32 for routing computation and discusses the trade-offs of different routing methods.

2. Expert load must be balanced. If a large number of tokens rush to a few experts, other devices will wait. The article compares methods such as auxiliary loss and bias adjustment without auxiliary loss. The control experiment with forced uniform routing is about 1.5 times faster, but it discards the learned routing, so it only illustrates the speed ceiling and cannot be directly treated as a training scheme.

3. Communication must not cancel out the computational gains. Once experts are distributed across different GPUs, each layer must dispatch tokens and collect results. In one performance analysis in the article, communication not hidden by computation accounted for about half of a step; in a case where the network was misconfigured and communication fell back to ordinary sockets, it was even about 25 times slower.

From training to deployment and reinforcement learning

The handbook recommends first checking hardware and networking, establishing a dense-model baseline, and then adding MoE settings one by one, rather than turning on all optimizations at once.

During deployment, reducing the number of experts selected each time can lower computation, but it will not automatically reduce the GPU memory occupied by model weights, and it may affect quality. Reinforcement learning has an additional problem: the inference engine that generates training data and the training engine may select different experts, causing inconsistent execution paths.

Source

Shao Meng