Baidu Open-Sources ERNIE 5.5 Flash Base, Setting SOTA for Same-Size Models Across Multiple Benchmarks

On November 5, Baidu officially open-sourced its latest base model, ERNIE 5.5 Flash Base. The model adopts a MoE architecture, with a total of 280 billion parameters, activating 14 billion parameters per token, and was pretrained on 33.5 trillion tokens. The pretraining context length reaches up to 128K, and the fine-tuning and inference stages support a context of up to 1 million tokens. It achieves SOTA performance among models of the same size on multiple benchmarks covering knowledge, reasoning, mathematics, code, multilingual capabilities, and long context, surpassing models such as DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base.

As large model applications extend from text question answering to long-document analysis, multi-turn interaction, and agent tasks, the volume of information models need to process continues to grow, and computational and memory overhead rises accordingly. Improving resource utilization efficiency while ensuring model capability has become an important topic in large model R&D.

English text is being prepared.

In the attention mechanism, ERNIE proposes Latent-Gated Sparse Attention, which combines hybrid sparse attention, low-rank gated attention, and virtual head attention. Among these, by combining HCA and MLA, the model lets different attention layers respectively take on the tasks of compressing historical information and fine-grained access to historical positions; Low-Rank Gated Attention uses existing low-dimensional representations to compute information gating, reducing the gating projection weight parameters by 75%; in addition, Virtual Head Attention (VHA) can enhance the model's expressiveness and training quality, and is fused into the attention operator during inference with minimal overhead. These designs reduce the resource overhead of long-context processing while preserving the model's ability to handle key information.

In expert routing, the model adopts Multi-Router Latent MoE, in which multiple independent routers jointly participate in expert selection, improving the correlation between routing scores and the actual utility of experts. Compared with traditional schemes that rely on a single router, this design aims to reduce the impact of a single scoring bias on expert selection while keeping the number of activated experts, expert computation, and communication volume unchanged. In terms of training objectives, the model introduces ECHO, which does not require an additional MTP (Multi-Token Prediction) module; instead, it reuses the last layer of the backbone, letting the backbone itself learn to predict subsequent tokens, thereby planning the sequence direction in advance and effectively improving reasoning and code capabilities. ERNIE 5.5 Flash adopts only single-step ECHO pretraining, and the reused last layer can also serve as a draft model for speculative decoding, demonstrating multi-step extrapolation capability: with the same single-step training, the quality of multi-step drafting is substantially better than that of an auxiliary MTP module.

Beyond the model architecture, innovation in large model infrastructure is also very important. ERNIE has optimized the distributed Muon optimizer and introduced FlashMask-4, an attention operator for large head dim, improving communication, memory utilization, and computational efficiency in large-scale training. Improvements in model capability depend not only on the algorithms themselves but are also closely related to whether the training system can efficiently support these designs.

Base models not only carry the general capabilities formed during pretraining but are also an important foundation for downstream model development and application innovation. Unlike application models aimed at end users, Base models provide developers and enterprises with room for further fine-tuning and customization, enabling them to build specialized models and applications suited to different scenarios based on their own business needs. Open base models can provide developers with more technical choices and lower the training cost and technical threshold of specialized model development.

At present, competition in large models is gradually moving toward a new stage of coordinated development of model capability, training efficiency, and scenario innovation. As technologies and applications such as reasoning and Agents accelerate their evolution, the industry's demand for base models is shifting from a single focus on capability improvement to more efficient training paradigms, more flexible capability expansion, and broader scenario adaptation needs. While improving the model's foundational capabilities and training and inference efficiency, ERNIE 5.5 Flash Base provides developers and the community with a more extensible technical foundation, driving model technology toward diverse capability exploration and application innovation.