Understand this story
Prime Intellect releases the Prime Mega MoE training kernel. This kernel builds on the Mega MoE kernel from DeepSeek's open-source DeepGEMM library by adding a fused and overlapped backward pass, making it usable for training rather than inference only. In the prime-rl trainer, a full training step for the Qwen3 MoE model is 32% to 34% faster than the baseline and 48% to 55% faster than TorchTitan; single-layer forward pass speedup reaches up to 3.11x (from 12.0 ms to 3.9 ms), but the backward pass for Qwen3-235B-A22B is only 1.24x faster. The kernel supports BF16, MXFP8 on Blackwell, and blockwise FP8 on Hopper, is released under Apache 2.0 with prime-rl, and is enabled via the mega_moe dispatch backend.
AI-generated from public reports
Latest development · Report
Prime Intellect Releases Prime Mega MoE Training Kernel, Up to 3.11x Faster in Forward PassDevelopments
1 confirmed developments
- Source:MarkTechPost
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 30
MarkTechPost
Prime Intellect Releases Prime Mega MoE Training Kernel, Up to 3.11x Faster in Forward Pass
Prime Intellect has released Prime Mega MoE, which builds on the Mega MoE kernel from DeepSeek's open-source DeepGEMM library by adding a fused and overlapped backward pass, making it usable for training rather than inference only. In the prime-rl trainer, the full training step for Qwen3 MoE models is 32% to 34% faster than the baseline and 48% to 55% faster than TorchTitan; the single-layer forward pass is up to 3.11x faster (from 12.0 ms to 3.9 ms), but the backward pass for Qwen3-235B-A22B is only 1.24x faster. The kernel supports BF16, MXFP8 on Blackwell, and blockwise FP8 on Hopper, is released with prime-rl under Apache 2.0, and is enabled through the mega_moe dispatch backend.