DeepSeek Open-Sources Inference Engine DeepInfer, Discloses That a Single Cluster Processes About 155 Trillion Tokens per Day

A single cluster processes about 155 trillion tokens per day, and inference optimization is a race for milliseconds.

Author: Zheng Jiamei

Editor: Cen Feng. Yesterday, DeepSeek open-sourced another major project on GitHub called DeepInfer. This time no new model was released; what was made available is the inference engine behind DeepSeek-V4.1-Flash, including core technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding. DeepSeek also disclosed a set of figures: a single cluster needs to process about 155 trillion tokens per day. With such enormous traffic, even if the GPU wastes a little resource in each round of computation, the accumulated waste is considerable. Currently, DeepInfer offers offline inference capabilities; components such as online KV Cache storage and cross-replica load balancing have not yet been fully disclosed. However, from the code already released, one can see how DeepSeek arranges computation, communication, and request scheduling in large-scale inference. Among these is an asynchronous scheduling mechanism called Zero-Overhead Batch Scheduler. The problem it solves is very specific: the GPU has clearly finished computation, yet it must stop and wait for the CPU to organize the previous round's generated results, update request states, and then arrange the next round of work. For requests that need to continuously generate thousands of tokens, this brief wait occurs repeatedly, and the GPU's actual utilization is also affected. DeepInfer lets the CPU arrange computation one round ahead, overlapping the originally serialized scheduling and model forward pass. But scheduling ahead means that when the next round of computation starts, the previous round's tokens may not yet have completed CPU post-processing, creating timing gaps in sequence length, KV Cache state, and request lifecycle. The technical difficulty of the entire mechanism lies precisely in this cross-round state management.

01

Rearranging Decode timing. Decode in traditional inference engines usually runs in a fixed order: the CPU selects requests, organizes batch and KV Cache metadata, the GPU executes Forward and Sampling, then the CPU reads the results, updates generation length and request states, and starts the next round of computation. As underlying kernels such as DeepGEMM and FlashMLA continue to be optimized, the GPU's single-round computation time keeps shrinking, and the time spent on CPU scheduling and result processing begins to account for a more noticeable proportion. Especially in the Decode stage, the model processes a limited number of tokens per round, and the frequent switching of execution control between CPU and GPU produces continuously accumulating gaps. Leiphone.com's Zero-Overhead Batch Scheduler adjusts this execution order. According to the public implementation, when the GPU processes the Nth round of Forward and Sampling, the CPU has already begun preparing the N+1th round batch, and uses the gaps during device computation to process the sampling results of an earlier round. Some work can be submitted to the GPU execution queue in advance, with dependencies maintained by CUDA Stream and Event. In the implementation, WorkerStepState is used to save cross-round execution state, including SequenceMetadata, SchedulerOutputs, SamplerOutput, and the CUDA Events corresponding to Forward and Sampling. In this way, the previous round's results can be temporarily retained, and the CPU can continue advancing subsequent scheduling without waiting for all of them to complete post-processing. LLMEngine connects these operations through Worker callbacks, spreading the CPU work originally concentrated between adjacent Decode Steps into the periods of GPU computation. For continuously generating requests, once the pipeline enters a steady state, the CPU and GPU can process different rounds separately, reducing device waiting caused by scheduling work. Of course, pipeline startup and teardown still require special handling. At the very beginning there are no earlier results to consume, and after all requests complete there may still be sampling outputs that have not been submitted. DeepInfer handles these in-flight results through a Flush mechanism, ensuring that after the final batch of computation completes, request states can be properly finalized. The benefit of this approach depends on how much CPU work can overlap with GPU computation. When CPU processing time exceeds the window that device computation can cover, waiting still occurs, so Zero-Overhead is closer to an execution scheduling goal.

02

How DeepInfer corrects state misalignment. When the CPU arranges the next round in advance, the previous round's sampling results may not yet be determined. Ordinary autoregressive decoding usually advances only one token, but with speculative decoding enabled, a request may accept multiple candidates consecutively, and different sequences may also differ in their actual advance length. Suppose a request has already cached 1000 tokens, and the current round may accept 3 candidates. When the CPU prepares the next round, it is not yet clear how many of these candidates will ultimately pass; if the old length is used directly to organize computation, Query Length, Key Length, and RoPE positions may be offset. For this reason, DeepInfer maintains total_inflight_tokens, recording tokens that have been issued but whose results have not yet been submitted. When the scheduler allocates computational resources, it incorporates this in-flight computation into sequence length and KV Cache capacity checks, and reserves the potentially needed cache space in advance, avoiding capacity shortages during subsequent GPU execution. After capacity reservation, there is still the problem of inconsistent request positions across two consecutive batches. Requests may finish, join, or be preempted, and the same sequence may also change position in different batches. DeepInfer therefore uses Remap mapping to correspond the current round's requests to the execution state saved from the previous round, while newly joined requests use a special marker. The GPU uses the mapping to find the previous round's actual sampling results, and then corrects parameters such as Query Length and Key Length through device-side kernels, so that the current round's attention computation can access the correct context range. For variable-length speculative verification, the number of tokens to process is also updated according to the actual acceptance results. Since CUDA Graphs usually run according to execution specifications captured in advance, the system also needs to distinguish between the capacity reserved by the computation graph and the actual effective sequence length. Device-side state correction reduces the need to frequently transfer small amounts of length data between CPU and GPU, and also allows dynamic requests to fit fixed-specification execution graphs. Another source of complexity comes from memory lifecycle. During synchronous execution, after a request completes, the CPU can usually directly release the relevant metadata and cache references. But in an asynchronous pipeline, although a request has already completed CPU post-processing, there may still be unfinished operations on the GPU referencing its memory. Leiphone.com's DeepInfer uses DeferredRelease to delay reclamation of fixed references to SequenceMetadata, RequestOutput, and related cache blocks. Resources to be released first enter a queue, and are then processed uniformly in the subsequent pre_sampler_sync stage. For resources involving GPU state copies, the system uses CUDA Events to confirm that operations are complete before releasing references, preventing the cache from being occupied by other requests prematurely. Request preemption must also follow the same state boundaries. If a sequence has already left the running set, then when previously submitted computation results return, it must not continue updating token counts or cache positions according to the old state, otherwise it may affect the sequence after rescheduling. These handling mechanisms allow advance scheduling to span multiple unfinished Steps, avoiding conflicts among sequence positions, cache allocation, and resource reclamation.

03

Ensure consistent execution pacing across multiple GPUs. The deployment scale of DeepSeek-V4.1-Flash further increases scheduling difficulty. In the official scheme, Prefill typically uses EP2 to EP8, while Decode uses wide expert parallelism of EP64 to EP128. Expanding the scale of expert parallelism can distribute the pressure of MoE weight reads, but GPUs participating in the same communication domain must adopt a consistent execution step pattern. For the NORMAL Engine that handles both Prefill and Decode, if some Ranks enter Prefill while other Ranks continue Decode, a collective communication mismatch may occur. DeepInfer adds an intra-group voting mechanism to the Scheduler to coordinate the execution modes of different Ranks. ALL mode requires the relevant Ranks to satisfy the conditions for executing Prefill, reducing idle devices that run along without tasks. ANY mode allows pending requests on some Ranks to trigger group-level Prefill, preventing small-scale requests from queuing for a long time. The two modes are controlled by their respective trigger intervals, reducing the extra overhead caused by frequent switching. There is another detail in the code: the presence of requests in the waiting queue does not mean that Prefill can be successfully scheduled in this round. A Rank may ultimately have no executable tasks because of insufficient KV Cache capacity or limited sequence slots. For this reason, after initially deciding to switch to Prefill, DeepInfer also performs a second round of confirmation. Each Rank completes actual scheduling, and then checks whether there is a valid Prefill within the group. If not, it returns to Decode; if some Ranks successfully schedule tasks, the other Ranks execute an empty Prefill Step to maintain a consistent communication order. When Zero-Overhead Scheduling is enabled, some intra-group scheduling coordination can also be executed through the CPU-side Data Parallel communication group, reducing interference from control communication with GPU computation. Beyond multi-GPU synchronization, MoE expert load also affects the execution time of the entire round. The Router may allocate more Tokens to a few hot experts, causing the corresponding GPU's workload to be higher than that of other devices; even if CPU scheduling has completed, computation still has to wait for the slower Rank. DeepInfer distributes this type of load through Expert-Parallel Load Balancing, retaining some redundant expert replicas on GPUs and rearranging expert distribution based on runtime statistics. The official Decode configuration uses a 1000-step statistics window and adjusts expert allocation every 10000 steps; the measured ratio of slow Rank load to average load is about 1.090. In this way, intra-group step pattern coordination is responsible for handling cross-device execution consistency, while expert load balancing alleviates the problem of some devices chronically slowing down the entire round of computation. Working together with asynchronous scheduling, the two allow DeepInfer to maintain relatively continuous Decode execution in large-scale MoE deployments.

04

Inference optimization begins to go deep into execution timing. DeepInfer's Zero-Overhead Batch Scheduler draws on existing asynchronous scheduling practices such as SGLang and NanoFlow. Its engineering characteristics are concentrated in cross-round state correction, in-flight Token accounting, KV Cache lifecycle management, and collaborative implementation with wide expert parallelism. At present, the official team has not separately published complete end-to-end speedup data for this scheduling mechanism relative to the synchronous scheme; DeepInfer's overall performance results also include the combined effects of underlying Kernels, model parallelism, and other execution optimizations. The currently public version is also mainly aimed at DeepSeek-V4.1-Flash, and some infrastructure required for a complete online service is still not open. As Coding Agent and long-chain reasoning tasks continue to increase, the number of rounds Decode needs to execute continues to grow. DeepInfer demonstrates a clear engineering route: reduce scheduling gaps through cross-round execution overlap, then use state correction and resource lifecycle management to maintain correctness, so that more GPU time is used for actual model computation. Reference link: https://github.com/deepseek-ai/DeepInfer/blob/main/README.md