Baidu Wenxin open-sources the ERNIE 5.5 Flash Base foundation model, with 280B total parameters
Baidu Wenxin has released and open-sourced ERNIE-5.5-Flash-Base, a sparse MoE language model with 280B total parameters and 14B activated per token, pretrained from scratch on 33.5T tokens. These weights have not undergone mid-training or post-training and can serve as a starting point for domain adaptation, instruction fine-tuning, and reinforcement learning research; fine-tuning and inference support a 1M-token context. The official statement says that the post-trained ERNIE 5.5 Flash will be released subsequently, and it is compared against DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base under the same framework across 30 benchmarks.
Why read this
AI score 57
The original text provides the parameters, training data volume, and a 30-benchmark comparison of the 280B sparse MoE foundation model, which can be used to assess its positioning as a fine-tuning starting point.
Topics
Body · AI translation
Baidu Wenxin Open-Sources ERNIE 5.5 Flash Base, an Efficient Foundation Model with 280B Total Parameters
2026-11-05 16:21 IT Home - Qincang (Intern)
English text is being prepared.
IT Home, November 5: Baidu's Wenxin large model today released ERNIE-5.5-Flash-Base, opening up the weights at the end of ERNIE 5.5 Flash pre-training.
This is a sparse MoE language model with 280B total parameters, 14B activated per token, pre-trained from scratch on 33.5T tokens; the maximum training context is 128K, and fine-tuning and inference support a 1M-token context.
The weights released this time have not undergone mid-training or post-training, and can serve as a common starting point for domain adaptation, instruction fine-tuning, and reinforcement learning research.
According to the official statement, the Base model is only the starting point. The post-trained ERNIE 5.5 Flash will be released subsequently.
ERNIE-5.5-Flash-Base was pre-trained from scratch on 33.5T tokens, using the Muon optimizer and FP8 mixed precision. The pre-training corpus covers a wide range of domains and languages, with focused development in three areas:
Web data: multilingual web crawling, high-fidelity text extraction, and model-driven multi-dimensional quality filtering.
Code data: repository and web code underwent non-code filtering, deduplication, and quality grading, and was supplemented with software evolution and execution-verification data such as commits, issues, pull requests, code reviews, automatically generated tests, and CI-verified fixes.
Long-document data: selected natural long documents, long sequences mixed with semantically similar retrieval distractor text, and synthetic retrieval, citation-based question answering, and multi-hop question answering data.
▲ Model architecture
According to the introduction, the training system and model architecture were co-designed. The optimizer uses distributed Muon: at the scale of thousands of GPUs, cross-machine communication for Dense parameters and MoE parameters is merged into a single collective communication, reducing cross-machine latency and jitter. The attention operator uses CuTe-DSL-based FlashMask-4, supporting SM90 and SM100 architectures, with head dim up to 576 (qk) / 512 (v), and can be directly used for computation of 512-dimensional wide heads in HCA layers.
IT Home attaches the evaluation data and open-source links:
Wenxin large model evaluated ERNIE-5.5-Flash-Base on 30 benchmarks, covering seven categories of capabilities: general, knowledge, science, reasoning, multilingual, code, and long context, and listed the results of same-scale Base models DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base; all four models are based on their respective publicly released checkpoints and use the same evaluation framework.
ERNIE-5.5-Flash-Base performs well across all capability dimensions; in terms of long context, although 37 of the 43 layers perform attention on sequences compressed 128 times, the 6 MLA-DSA layers are sufficient to support fine-grained retrieval. The complete table is available in the Hugging Face model card.
Advertising statement: External redirect links contained in the article (including but not limited to hyperlinks, QR codes, passcodes, etc.) are used to convey more information and save selection time; the results are for reference only. All articles on IT Home that contain external links include this statement.