NVIDIA 如何用投机解码协同设计 AI 模型以加速 LLM 推理

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Related Resources

Developer Tools & Techniques

English 中文

Sep 02, 2026

Speculative decoding accelerates LLM inference by having a small draft model propose multiple tokens that a larger target model verifies in parallel, reducing decoding iterations while preserving output accuracy. Five guidelines help select the optimal draft length and mechanism across the Pareto frontier: push GEMMs into the compute-bound region without increasing KV cache pressure; when attention dominates, set draft length to 128/G - 1; align larger draft lengths with attention kernel tile boundaries of 128; at very low latency, increase draft length only while acceptance gains justify the added draft cost; and choose the draft mechanism by balancing acceptance length, draft overhead, and training and deployment cost. Draft mechanisms compared include external draft models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods, each with different tradeoffs in training cost, serve-time memory, and speculation cost per draft tokens. SPEED-Bench, a speculative decoding benchmark developed by NVIDIA, provides realistic workloads for measuring acceptance length across task domains such as coding and summarization. Ready-to-run training examples for EAGLE-3, DFlash, and DSpark are available in NVIDIA/Model-Optimizer , including fine-tuning and quantization workflows demonstrated on Nemotron 3.5 Lightning. Next Steps Explore the NVIDIA/Model-Optimizer repository for training examples and quantization workflows. Benchmark acceptance length on SPEED-Bench with realistic prompts and diverse task domains. Quantify draft overhead using NVIDIA TensorRT LLM for high-performance inference.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding and offers five guidelines for selecting draft length and draft mechanism across the Pareto frontier.

For a discussion of how model design choices impact both throughput and interactivity without sacrificing accuracy, see AI Model Co-Design: Hardware-Friendly LLM Design (Part 1). For an explanation of how group size (query heads per KV head), head dimension, and sequence length shape the performance of dense attention, see Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference (Part 2).

Speculative decoding is a technique for accelerating the autoregressive decoding phase of LLM inference by predicting multiple tokens per iteration. A small draft model first predicts several likely next tokens. These tokens are then verified in parallel with a single pass through the larger target model.

Figure 1. Speculative decoding moves LLM decoding toward a more compute-bound region, shifting the throughput-interactivity Pareto frontier up and to the right. As interactivity increases toward the right of the curve, batch size decreases, and GEMMs account for a larger share of runtime than attention

This approach reduces the total number of decoding iterations while increasing the arithmetic intensity of the target model, without requiring higher concurrency. The target model accepts the proposed tokens in sequence until it encounters the first mismatch. The next prediction cycle then resumes from that position. Because only tokens accepted by the target model are retained, speculative decoding produces the same output sequence as standard decoding, unless the acceptance criteria are deliberately relaxed. Figure 2 illustrates this decoding flow.

The draft length (\(D\)) is the number of proposed tokens per target iteration. The acceptance length (\(AL\)) is the number of tokens produced (that is, accepted) per target iteration. \(AL\) ranges from \(1\) to (\(1 + D\)) because the target can always produce one new ground-truth token in addition to the accepted draft tokens.

The speedup from speculative decoding can be quantified as the ratio of time taken by the target model to sequentially generate \(AL\) tokens to the time taken to verify D tokens in parallel, while also accounting for the latency overhead of drafting \(D\) tokens:

\(\mathrm{speedup} = \frac{T_{\mathrm{verif}}(B) \times \mathrm{AL}}{T_{\mathrm{verif}}\left(B \times (1 + D)\right) + T_{\mathrm{draft}}(B, D)}\)

where \(B\) is the batch size, \(T_{\mathrm{verif}}(x)\) is the target verification time for \(x\) tokens, and \(T_{\mathrm{draft}}(b, y)\) is the time needed to generate a draft of length \(y\) for batch size \(b\). It is evident that to maximize speedup, it’s necessary to find the optimal \(\left(D, AL, T_{\mathrm{draft}}\right)\) tuple.

Figure 2. The draft-then-verify flow of speculative decoding. A smaller draft model proposes \(D\) tokens, and the target model verifies them together in one pass

Selecting the optimal draft length

For simplicity, if we ignore draft model latency, speculation provides a speedup when:

\(\frac{T_{\mathrm{verif}}\left(B \times (1 + D)\right)}{T_{\mathrm{verif}}(B)} \frac{128}{G} – 1\), prefer values where \(G \times (1 + D)\) is a multiple of 128 to avoid tile underutilization.

The relative weight of Guideline 1 compared to Guidelines 2 and 3 depends on the percentage of runtime spent in FFNs compared to attention at the preferred operating point. Communication also adds verification cost as D increases, although compute and communication overlap can mitigate the overhead.

Moving towards the far right of the Pareto curve, \(B\) becomes very small. Here, fixed kernel setup and post-processing overheads dominate for both compute and communication kernels. These fixed costs don’t grow significantly with the number of verified tokens, keeping verification overhead largely invariant with the number of draft tokens.

While MoEs do see more experts activated with increasing draft length, a combination of model sharding strategies and efficient kernels such as Grouped GEMMs can keep this overhead low. Larger draft lengths can therefore help in the low latency region, as long as acceptance remains high.

At very low latency, the number of sequential kernel launches determines workload latency. Because kernel launches scale linearly with layer count, for an autoregressive draft model with a similar layer structure to the target, the speedup can be approximated as:

\(\mathrm{speedup} = \frac{L_{\mathrm{target}} \times \mathrm{AL}}{L_{\mathrm{target}} + D \times L_{\mathrm{draft}}}\)

where \(L_{\mathrm{target}}\) and \(L_{\mathrm{draft}}\) are the layer counts in the target and draft models, respectively. Defining a constant draft_depth_ratio, \(\rho = \frac{L_{\mathrm{draft}}}{L_{\mathrm{target}}}\):

\(\mathrm{speedup} = \frac{\mathrm{AL}}{1 + \rho D}\) and draft overhead, \(O_d = \rho D\)

In other words, increasing \(D\) helps only while the gain in \(AL\) is large enough to offset the draft overhead.

Guideline 4: At very low latency, increase \(D\) only while the gain in \(AL\) justifies the added draft cost.

Selecting \(D\) tells you how many tokens to speculate. Next, decide how to generate these tokens to maximize speedup.

A multitude of techniques have been proposed over the years, with different tradeoffs in training, parameter, and runtime costs. External drafting suggests using a small standalone LLM, while MTP, EAGLE-3, DFlash, and DSpark use auxiliary layers combined with information from the target model to predict tokens. Suffix and n-gram methods are model-free and instead reuse patterns already seen in the token stream.

Table 1 compares key drafting methods, including how they generate tokens, their serve-time memory, and their draft overhead.

To quantify the tradeoff between \(AL\) and \(O_d\), begin by observing how \(AL\) scales with \(D\).

Figure 6 shows how AL changes as D increases on SPEED-Bench , with Qwen 3.5 122B A10B as the target. SPEED-Bench is a speculative decoding benchmark developed by NVIDIA with the goal of representing realistic production workloads. It covers multiple task domains such as coding and summarization and has a variety of splits at different input sequence lengths. We recommend SPEED-Bench for \(AL\) comparisons.

On the 32K split, Qwen 3.5 35B A3B reaches an \(AL\) of 6 at \(D = 9\), while the 4B draft reaches an AL above 5. MTP and DFlash ALs level off as \(D\) grows. N-gram has lower acceptance on this workload and is better suited to workloads with repeated token patterns.

Figure 6. \(AL\) versus \(D\) for different draft mechanisms on SPEED-Bench, throughput 32K split, with Qwen 3.5 122B A10B as the target

Higher \(AL\) does not equal higher speedup. You also need to consider how much it costs to generate the draft.

As shown in Figure 6, all external drafts reach higher \(AL\) than other draft methods at \(D > 3\). The Qwen 3.5 122B MTP is much smaller, with 2.5B total parameters and fewer than 150M active parameters. At \(D = 3\), a larger external draft may not be worth the added cost. At \(D = 21\), its higher AL may make that cost worthwhile, assuming an efficient serving strategy.

Meanwhile, DFlash \(AL\)s level off relatively quickly. However, both DFlash and DSpark generate \(D\) draft tokens in parallel, lowering \(O_d\). For comparison, at minimum latency and with \(D=11\), a one-layer MTP head takes 11 steps, leading to \(O_d^{\mathrm{MTP}} = \frac{11}{L_{\mathrm{target}}}\), while a five-layer DFlash head generates the draft in one pass, giving \(O_d^{\mathrm{DFlash}} = \frac{5}{L_{\mathrm{target}}}\).

For a large target model with many layers, both overheads are negligible, but as \(L_\mathrm{target}\) decreases, the draft overhead becomes significant. For small models at low latency, therefore, DFlash or DSpark can be the optimal choice even with lower \(AL\).

To navigate the tradeoff between \(AL\) and draft latency, it is crucial to benchmark both accurately. In the context of speculative decoding, this means benchmarking with realistic prompts and covering a wide set of task domains. We recommend SPEED-Bench for \(AL\) measurements and high performance inference frameworks such as NVIDIA TensorRT LLM to quantify draft overhead.

Beyond inference performance, draft training scope and costs deserve consideration. While MTP must be co-trained with the target model, EAGLE, DFlash, and DSpark can be added to the final model checkpoint. Similarly, a continuum of options exists for external draft model creation. Training from scratch and distillation from the target yield the highest ALs, while adapting an existing model to the desired target through cross-model adaptation techniques can significantly reduce training costs at the expense of \(AL\).

Fine-tuning the target model can change its output distribution and hidden representations. Learned drafters trained for a particular target checkpoint can therefore lose acceptance even when the target model improves. After changing the target, remeasure \(AL\) on representative workloads. The required adaptation depends on the draft mechanism.

MTP is part of the target model and should continue to be trained during fine-tuning or be realigned in a dedicated follow-up stage. Target-attached drafters such as EAGLE-3, DFlash, and DSpark use the target’s hidden states and will need to be adapted to the updated checkpoint with additional training.

External draft models do not consume target hidden states, but they still approximate the target’s output distribution and may require fine-tuning or distillation. Suffix and n-gram methods have no learned drafter and therefore require no retraining, although their effectiveness still depends on how much repetition appears in the deployed workload.

Guideline 5: Choose the draft mechanism that gives the best decode speedup for your workload and hardware, considering \(AL\), draft latency, and training and deployment cost.

Use the five guidelines summarized below as a checklist for selecting \(D\) and draft mechanism across the Pareto frontier.

For post-trained drafters, ready-to-run training examples are available in NVIDIA/Model-Optimizer for EAGLE-3, DFlash, and DSpark. See how we did this for NVIDIA Nemotron 3.5 Lightning: fine-tune DSpark , then quantize it to FP8 or NVFP4 . Use the examples as starting points, then validate AL and end-to-end speed on your own model, workload, and hardware.

This post is a cross-team NVIDIA effort. We are grateful to Bhargava Gopireddy, Ritika Borkar, Dor Tsur, Andrii Skliar, Benjamin Chislett, Yaniv Galron, Talor Abramovich, Yoav Miron, Rabeeh Karimi Mahabadi, Roger Waleffe, Udi Karpas, Ran Zilberstein, Brian Pharris, and Eduardo Alvarez for their help.