DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day in a single cluster

DeepSeek has open-sourced DeepInfer, its self-developed inference framework that supports DeepSeek-V4.1-Flash, and published production data: a single cluster processes 155 trillion tokens per day, 96% input cache hit rate, 400ms first-token latency, and 230 tokens/s generation speed. The framework is customized based on vLLM and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, with target platforms NVIDIA SM100 and Huawei Ascend 950. DeepSeek also states that components such as KV cache management and load balancing are not yet fully open-sourced, while the vLLM team says that under a matching GB200 configuration, decoding throughput is within 6% of the results published by DeepInfer.