Understand this story
English text is being prepared.
AI-generated from public reports
Latest development · Report
Hugging Face releases MoE training handbook, investing about 800,000 GPU hours in experimentsDevelopments
2 confirmed developments
- Source:Shao Meng
- Source:English text is being prepared.
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 6
Shao Meng
Hugging Face releases MoE training handbook, investing about 800,000 GPU hours in experiments
The Hugging Face team invested about 800,000 GPU hours in experiments and compiled the results into a free, open "Ultimate MoE Training Handbook," roughly 70,000 words long, with 55 interactive diagrams, runnable code, and real experimental data. The handbook covers questions such as choosing the number and size of experts, cross-GPU communication slowing down training, routing and load-balancing settings, and whether to convert a dense model or train from scratch. The example in the text uses 64 experts with 8 selected each time and a width of 1024 per expert; the entire example model has about 6.86 billion total parameters and 1.22 billion activated parameters. In one performance analysis, communication not hidden by computation accounted for about half of a step, and a case where communication fell back to ordinary sockets was about 25 times slower.
Ma Dongxi NLP
Thomas Wolf releases "The Ultimate MoE Training Handbook"
Thomas Wolf has released the free and open-source "The Ultimate MoE Training Handbook." Based on 800k GPU hours of experiments, it systematically explains how to train Mixture-of-Experts models. The handbook is about 70,000 words long and includes 55 interactive charts and runnable code. It addresses common questions faced by training teams, such as the number and size of experts, all-to-all communication overhead, router choice (softmax, sigmoid, or sqrt(softplus)), aux loss and DeepSeek's loss-free bias, and dense checkpoint upcycle. He points out that 18 of the 20 best open-source models today are MoE.
Thomas Wolf
Thomas Wolf releases the open-source training handbook "The Ultimate MoE Training Handbook"
Thomas Wolf has released the free, open-source "The Ultimate MoE Training Handbook," compiled from 800,000 GPU hours of experiments and organized around the problems training teams commonly encounter, including the number and size of experts, all-to-all communication overhead, the choice of routing function, auxiliary losses and DeepSeek's loss-free bias, upgrading dense checkpoints, and reducing the number of experts per token at inference time. The handbook is about 70,000 words long, contains 55 interactive charts and runnable code, and includes data from the author's own experimental runs.
Heat trend for this story
- Current
- 5
- Period peak
- 25 Nov 6 21:00
- 24-hour change
- -63%
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.