NVIDIA 解析 Dense 与 MoE 模型：活跃参数、吞吐量与选型指南

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each

Nemotron 3.5 Lightning uses a Mixture-of-Experts architecture that activates only 3B of its 30B total parameters per token. MoE models decouple memory cost from compute cost by loading all experts into VRAM while activating only a subset per token, unlike dense models where both scale together. At equal total parameter counts, MoE models such as Nemotron 3.5 Lightning achieve higher token throughput than dense models such as Gemma 4 31B , though the latency advantage narrows at high concurrency. Fine-tuning MoE models requires care to avoid router imbalance, and quantization affects router and recurrent-projection layers differently than dense-model components. Deployment choice depends on memory budget, concurrency needs, fine-tuning plans, and quantization behavior rather than raw parameter count alone. Next Steps Try Nemotron 3.5 Lightning on build.nvidia.com to evaluate the model interactively. Download the Nemotron 3.5 Lightning weights from Hugging Face for local deployment. Access the model via OpenRouter for API-based inference.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the answer: It uses a Mixture-of-Experts ( MoE ) architecture that selects only a subset of its parameters for each token .

There are two dominant model architectures: Dense model and MoE. How a model organizes its parameters matters as much as how many it has. The right choice affects throughput, memory cost, and serving complexity more than raw parameter count does. Therefore, choosing between them comes down to your deployment constraints.

Think of the difference like two engines with the same total displacement. One fires every cylinder on every cycle; the other activates only the cylinders it needs.

This post explains:

In simple terms, dense and MoE models differ in how they use their parameters. A dense model activates all its parameters for every token. An MoE model stores multiple expert networks but routes each token through only a selected subset.

Dense models typically favor simpler, more predictable deployment, while MoE models can deliver greater capacity and throughput when memory and serving complexity are manageable.

In a dense model every parameter participates in every forward pass. All 27B parameters of a 27B model fire for every token, through a single shared feed-forward network (FFN) block per decoder layer.

An MoE model replaces that single shared FFN with multiple FFN blocks , also known as experts. Through an internal routing process, incoming tokens are processed through a small subset of experts rather than every single parameter. Structurally, inside each decoder layer where a dense model would have one FFN, an MoE layer has multiple (e.g. 8, 64, or 128). A learned gate network , usually called a router network, sits in front of all of them and assigns each incoming token to the top k scoring experts. The selected FFN blocks are the ones that run for that token while the rest are skipped for that particular layer. Most modern MoEs though such as Mistral Small 4 also run one “shared” expert that every token is routed to regardless.

For MoE models, the routing decision made is separate for every layer, meaning a token doesn’t get routed to an expert and stays there for the rest of the computations. At each decoder layer it gets rerouted based on what the token represents at that point in the network. These experts at every layer aren’t experts in the traditional sense where they specialize in a subject, but rather their specializations lie primarily in syntax and token-type patterns (punctuation, numbers, etc), though this varies by architecture or training methods.

While the router mechanism decides which FFN blocks are skipped at each decoding layer, the tokens still pass through the full attention mechanism as normal. When a model card says “3B active parameters,” it includes both the attention and embedding weights for every token along with the selected FFN weights.

There are variants to MoE as well, one of them being the NVIDIA model card for Nemotron 3.5 Lightning specifying a Mamba-2 + MoE + Attention hybrid . The Mamba-2 layers stand in for attention in most layers, carrying a constant-size recurrent state rather than a growing KV cache. That changes the memory profile at long context in a way sparsity alone doesn’t account for.

Rather than Lightning making these routing decisions using the full model’s breadth, it compresses them into a smaller space first to make the routing decision cheaper for the model. Figure 1, below, describes the general transformer-MoE case. Note that an MoE model is a type of sparse model.

Figure 1. A dense model and sparse model decoder block with different routing mechanisms ( Source: Medium )

Which is faster: Dense or MoE models?

MoE models are often faster for token throughput because they activate only a subset of feed-forward parameters for each token. Dense models activate the full network, but can offer simpler serving and more predictable latency.

At high concurrency, routing and memory movement can narrow MoE’s advantage. Results also depend on hardware, precision, inference framework, and model design. For example, Nemotron 3.5 Lightning has Mamba-2 layers and speculative decoding.

The difference in performance between the two comes down to two main things. For one, at equal total parameter count the skipped FFN blocks make MoE faster. The more consequential difference between the two is the fact that MoE decouples memory (total params → VRAM to host) from compute (active params → FLOPs per token).

In a dense model hosting and inference costs scale together, and MoE breaks that link. With MoE, VRAM is paid up front. When every expert loads into memory the compute is paid per token and scales only with the experts that fire. Idle experts cost nothing to run and while still paying the required memory to store it, making that shift from variable per-token compute to fixed memory the main tradeoff.

At batch size 1 decoding is bound by available memory rather than compute, and this is where MoE performs well since it reads fewer weight bytes per token. As the batch size grows, tokens collectively end up using most of the experts in the network and that advantage narrows while the reduction in work per token persists. MoE holds a throughput advantage across batch sizes, but its latency margin compared to a well-optimized dense model compresses at high concurrency.

Modern inference frameworks typically handle routing without discarding tokens, though all experts must live in the GPU memory simultaneously. This leaves less room for the KV cache than a dense model of comparable size would typically have.

Table 1, below, shows a direct comparison. At equal total parameter size, sparsity shows up plainly in serving throughput: Gemma 4 31B and Nemotron 3.5 Lightning are both ~30B total, yet across NVIDIA GPU providers their output-speed ranges don’t overlap. Activating 3B parameters per token is a major reason, though not the only one. Lightning’s Mamba-2 layers and its speculative decoding contribute independently of the hybrid MoE architecture.

The tradeoffs for both can be observed; for example, Lightning delivers four to five times Qwen3.8-27B’s output speed at a fourteenth of the price, and scores less than half as well on general capability. That is right for an agentic execution layer running well-specified steps at volume, and wrong where one hard reasoning pass decides the outcome.

When it comes to deciding which model to use, it depends on which one will best serve your deployment context.

There are several factors to consider:

Dense and MoE are two answers to one tradeoff: capability per parameter against compute per token. Dense keeps everything simple and everything active, which makes it easier to fine-tune and to serve. MoE buys throughput with memory, and pays for it in serving complexity.

For physical AI workloads, AgiBot GO-1 and Tencent Hy-Embodied-VLM-1.0 are popular options in the ecosystem.