Shanghai Jiao Tong University's Deng Zhijie: How Diffusion Language Models Can Balance Generation Speed and Quality
At CNCC 2026, Deng Zhijie, an associate professor at Shanghai Jiao Tong University, systematically reviewed acceleration methods for diffusion language models, from the foundational LLaDA and Dream approaches to cross-block parallelism, post-training, and inference system optimization. Among them, Fast-dLLM requires no additional training and, under the settings of LLaDA-Instruct, GSM8K 5-shot, a single A100, and batch size 1, increased speed from 6.7 to 54.4 TPS, while accuracy changed from 79.3% to 78.5%. He also introduced explorations of related methods in streaming video, speech restoration, multimodal, and agent applications.
Body · AI translation
From basic approaches to system optimization, this talk explains how to turn parallel potential into actual speed.
Author: Zhang Lu
Editor: Zhang Chi
Writing is often accompanied by overall conception, local refinement, and repeated revision. The word choice in a sentence depends both on the preceding text and possibly on the following text. From this starting point, can the generation process of language models also unfold simultaneously at multiple positions and then gradually complete the text?
At the recent CNCC 2026, Deng Zhijie, an associate professor at Shanghai Jiao Tong University, gave a talk titled "Diffusion Large Language Model Methods and Applications," centering on a core question: can diffusion language models generate faster while ensuring generation quality?
From the basic approaches of LLaDA and Dream to cross-block parallel generation, post-training, and inference system optimization, Deng Zhijie reviewed various methods for turning parallel potential into actual speed.
He emphasized that evaluating efficiency requires looking at both how many tokens can be generated per round and the time cost per round, and that generation quality should be assessed together with answer accuracy and code pass rates. At the end of the talk, he introduced explorations of related methods in streaming video, speech restoration, multimodality, and agents. Leiphone
The following is the main content of the talk. 01
Language generation can unfold at multiple positions
When we write a book, we often write a first draft and then rewrite it repeatedly. When revising earlier content, we connect it with later chapters; deliberating whether to use "the monk pushes the moonlit door" or "the monk knocks on the moonlit door" also requires context. This shows that creating text does not necessarily strictly follow a left-to-right order.
Diffusion language models put this idea into practice: predicting multiple missing positions simultaneously and gradually completing the text. This talk mainly discusses how to increase generation speed as much as possible while ensuring generation quality.
Autoregressive models use the chain rule to decompose the joint probability distribution into a series of conditional distributions, and standard decoding predicts the next token, the basic unit of text, in sequence. The masked diffusion model highlighted here first randomly masks different positions in the sequence independently, with the masking ratio varying over time steps, and then learns the reverse restoration process: predicting the missing content based on the masked sequence.
At inference time, the model predicts all masked positions in each round but commits only some of them, leaving the rest for subsequent iterations. For example, six tokens can be completed in three rounds. However, fewer rounds do not mean actual speed increases proportionally, because the computational cost of each round must also be considered.
LLaDA extended this approach to the 8B parameter scale. During pre-training, it randomly masks the complete sequence; during supervised fine-tuning, it masks only the answer portion. It also demonstrated the trend of generation capability scaling with training compute.
Dream takes another approach: reusing the weights of existing autoregressive models and then continuing training with diffusion. Two aspects need to be adjusted here: attention changes from a causal structure to a bidirectional structure, and the learning task changes from predicting the next token to restoring masked content. Dream also retains a structure that predicts positions offset by one, to better reuse the capabilities of autoregressive models. Both pre-training from scratch and converting from existing models can build diffusion language models with good performance. 02
From intra-block parallelism to cross-block parallelism
To evaluate efficiency, we can start with a simple relationship: actual speed is approximately equal to the number of effectively committed tokens per round divided by the time cost per round. The former is measured by TPF, i.e., how many tokens are actually committed per forward computation; actual generation speed is measured by TPS, i.e., how many tokens are generated per second. At the same time, quality metrics such as answer accuracy and code pass rates must also be observed.
Early diffusion language models had two practical problems: first, it was difficult to reuse the KV cache, requiring repeated computation of context; second, the generation window size was fixed, making it difficult to flexibly output long text. If only two or three tokens can be accepted per round while the entire window must be recomputed, the computational cost keeps accumulating.
Block Diffusion adopts a "diffusion within blocks, autoregression across blocks" approach. Within a block, generation can be parallel; once complete, its KV state is cached, and then the next block is generated. This both reuses computation and extends output length by appending blocks.
But it limits the scope of parallelism to within a block. For example, a pair of parentheses far apart in code cannot be generated simultaneously if they belong to different blocks. Therefore, we hope that while the previous block is not yet complete, the next block can also start generating at the same time.
D2F (Discrete Diffusion Forcing) introduces the idea of Diffusion Forcing into language models: earlier blocks have lower noise and more complete information, later blocks have higher noise, and multiple blocks are denoised simultaneously. As the earlier content gradually becomes determined, later generation also progresses step by step.
During training, we use a teacher model with bidirectional context to guide a student model that adopts block-wise causal attention. The student uses bidirectional information within blocks and maintains causal relationships across blocks, so that completed preceding blocks can enter the KV cache.
At inference time, as the generation progress of the current block advances, the system appends new blocks; completed blocks enter the cache, and subsequent blocks continue generating.
The gain from this approach comes from two aspects: multiple blocks generate simultaneously, increasing the amount committed per round; and when caching completed blocks, subsequent blocks can continue to be decoded, reducing the situations where a separate forward computation is occupied solely for writing to the cache. 03
Adapting training to actual multi-block inference
Further analysis reveals that D2F still suffers from a mismatch between training and inference. During training, the entire response is split into multiple blocks, and the noise typically increases gradually; during actual inference, usually only two or three blocks are active at the same time, and the noise gap between adjacent blocks can be large.
To address this, we propose MultiBD, that is, multi-block diffusion. The training design must cover the window sizes, block positions, and noise intervals encountered in actual inference. For example, when generating two, three, or four blocks simultaneously, a block may be at different positions and have different generation progress. By simulating these combinations, the model can better adapt to the states at inference time.
Once the algorithmic parallelism increases, a corresponding inference engine is also needed. During multi-block generation, the number of active blocks may change each round, and the computation graph changes accordingly. We set up a fixed-size Block Buffer, using fixed physical slots to accommodate dynamically running blocks: completed blocks enter the cache, and new blocks reuse idle slots, thereby reducing the overhead of dynamic scheduling.
Under a test setup with two H100s, a single sample, and a tensor parallelism degree of 2, MBD-LLaDA2-Mini reaches about 746 TPS. Here, the output per step and the time per step must still be considered together; actual speedup cannot be judged based on TPF alone. 04
Improving the quality-speed trade-off with post-training
Even with multi-block generation, there is still a trade-off between quality and speed. Lowering the decoding acceptance threshold and accepting more tokens per round can speed up generation, but may also harm quality. This is especially true in the early stage, when the context is not yet sufficient, and committing too much content at once can easily amplify misjudgments.
One type of method is Consistency Force, or CForce: using later, more sufficient predictions to guide earlier predictions.
Specifically, first collect the model's own decoding trajectories, then divide them into stages according to the mask ratio, and form supervision pairs from adjacent stages, so that early predictions approach later predictions. For models with editing capability, both the positions to be filled and the positions that have been generated and then modified can participate in learning. At the same time, a distribution constraint related to the target confidence is used, and cross-entropy supervision is added to help the model learn to commit tokens correctly.
Under the same decoding threshold and block length, after CForce training, LLaDA2.1-mini's average score on four tasks increased from 85.57 to 86.41, and the average TPF increased from 6.94 to 9.08. In a system test with two H20s, a tensor parallelism degree of 2, and a single sample, the speeds on GSM8K and HumanEval were about 1232 and 1353 TPS, respectively.
Another type of method is reinforcement learning. For diffusion language models, the same question can have different generation orders. These trajectories may differ not only in the final answer but also in generation parallelism. Therefore, the reward can consider both "whether the answer is correct" and "whether the generation is fast."
Our LightningRL standardizes the correctness reward and the TPF reward within groups separately, then combines them for updates, while adding a token-level negative log-likelihood loss to prevent training from deviating. During sampling, it is also necessary to explore trajectories with different parallelism as much as possible and preserve the exploration space that can yield correct answers.
On two math tasks and two code tasks, LightningRL reaches an average of 7.32 TPF, and 11.10 on MBPP. In a comparison with TPF fixed at 7, GSM8K accuracy increased from 71.2% for the original SDAR to 87.5%. 05
After fixing the model, how to further accelerate
In addition to training, the inference algorithm can also be improved. Here we introduce Fast-dLLM and LoPA.
Fast-dLLM requires no additional training and mainly combines two strategies. The first is approximate KV caching. For bidirectional attention models, changes in context affect the states at other positions, so reusing old states is an approximation. By caching the prefix and the not-yet-generated masked suffix, repeated computation can be reduced.
The second is confidence-based decoding: instead of presetting a fixed number of tokens to accept each round, it accepts positions whose prediction confidence exceeds a threshold. Caching mainly reduces the time per round, while confidence decoding mainly increases the number of commits per round.
In experiments with LLaDA-Instruct, GSM8K 5-shot, a generation limit of 256, a single A100, and a batch size of 1, combining the two increased speed from 6.7 to 54.4 TPS, while accuracy changed from 79.3% to 78.5%.
LoPA further considers: when currently selecting a position to decode, how can we try to ensure that the next step is more likely to produce a large number of high-confidence predictions, that is, higher parallelism in the next decoding step, thereby accelerating the entire inference process? In other words, this actually requires looking one step ahead during decoding.
But pursuing only future confidence can also cause problems. A naive lookahead strategy may select a path that generates faster but has poor content quality. Therefore, for LoPA we retain the base path based on current confidence, then construct multiple lookahead candidates, compare in parallel their confidence at the remaining masked positions, and select the branch more conducive to continued parallel decoding, while reusing the prediction results obtained from verification.
Adding exploration branches can usually increase potential parallelism, but the gains gradually saturate. At the system level, we distribute the verification of different candidates across multiple GPUs for parallel execution, then synchronize the selected state and cache. In the MBPP test, combined with LoPA's distributed inference engine, the generation speed for a single-sample request reached 1073.9 token/s. 06
From text to video, speech, and agents
The first application direction is streaming video generation. The idea of multi-block parallelism is closely related to video generation, and video can usually use a fixed-size running window. We extend this idea to WaveForcing, placing different video blocks at different denoising stages and continuously outputting once the pipeline is filled.
This requires both training that adapts to the noise states of multiple blocks and system optimization. When different GPUs process different blocks, KV communication can be overlapped with computation; VAE decoding must also run in parallel with the generation process to avoid becoming an overall bottleneck.
In tests, the end-to-end speed reached about 117 frames/s, and subsequent system optimization further reached 150 frames/s, while balancing video quality, dynamics, and long-video stability.
The second direction is speech restoration. Faced with noise, reverberation, bandwidth limitations, and packet loss, the model must both restore natural-sounding audio and preserve "what was said" and "who is speaking." Diffusion language models can restore damaged semantics by leveraging the context of the entire sentence, and can also exploit the language priors from text pretraining.
TIMUSE processes this in two steps: first, it encodes through Whisper and uses a text diffusion language model to restore semantic tokens; then, combining the acoustic information of the degraded speech, it reconstructs the audio through conditional flow matching and a vocoder. Experiments show that masked generation and text pretraining help improve content accuracy while largely preserving audio quality and timbre.
There are also multimodal and agent directions. LLaDA-o explores combining text diffusion with the generation of modalities such as images, coordinating the representations and generation processes of different modalities through bidirectional attention between understanding and generation branches.
In agent scenarios, the model needs to call tools, receive feedback, and then continue generating. LLaDA 2.2 also explores editing operations such as deletion, insertion, and regeneration. Supervised fine-tuning with corresponding corpora can enable diffusion language models to learn these capabilities. This is also an application direction worth further exploring in the future. 07
On-site Q&A
▎Q: Unlike images, text does not have a fixed size. Can diffusion language models flexibly determine the generation length?
A: Yes, now they can. Early models usually had a fixed window, such as 512 tokens; if the answer exceeded the window, it might be truncated. If the answer is completed early, an EOS, i.e., an end marker, can be generated, and there is no requirement to fill the window.
With block-based generation, if there is no EOS at the end of a block, the next block can be appended. The specific way to stop based on EOS depends on the distribution during model training and the inference strategy; different models do not have completely unified rules.
The international robotics conference CoRL 2026 is about to be held. Researchers, students, and readers interested in the latest AI advances are welcome to join our communication group.
Here, you can meet peers, exchange research ideas, share admissions information, research collaborations, meal invitations, submissions, team formation for attending the conference, and more, and find potential partners.
During the conference, we will also continue to provide on-site reporting updates, interpretations of guest talks, overviews of frontier directions, and highlights from forums.
See you at the venue, and see you in the group.
? Group entry link: Scan the QR code to join the group or add WeChat szhsn610, with the note: CoRL + affiliation/school + name + direction.
Without authorization from "GAIR DAO", reproduction in any form on websites, forums, or communities is strictly prohibited!
For WeChat official account reproduction, please first leave a message on the "GAIR DAO" backend to obtain authorization; when reproducing, indicate the source and insert the official account card.
AI Trends