Search “Reasoning”

English translation is being prepared.

English translation is being prepared.

DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day in a single cluster

DeepSeek has open-sourced DeepInfer, its self-developed inference framework that supports DeepSeek-V4.1-Flash, and published production data: a single cluster processes 155 trillion tokens per day, 96% input cache hit rate, 400ms first-token latency, and 230 tokens/s generation speed. The framework is customized based on vLLM and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, with target platforms NVIDIA SM100 and Huawei Ascend 950. DeepSeek also states that components such as KV cache management and load balancing are not yet fully open-sourced, while the vLLM team says that under a matching GB200 configuration, decoding throughput is within 6% of the results published by DeepInfer.

DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day on a single cluster

DeepSeek has open-sourced DeepInfer, the production inference framework supporting V4.1 Flash, and published production data including 155 trillion tokens processed per day on a single cluster, a 96% input cache hit rate, an average 400ms time to first token, and 230 tokens/s generation. The framework is customized based on vLLM and uses the Apache 2.0 license, supports distributed inference, speculative decoding, SWA Bounded Replay and structured output, and targets NVIDIA SM100 and Huawei Ascend 950. DeepSeek says components such as KV cache management and load balancing are not fully open-sourced because they are coupled with internal infrastructure, while the vLLM team reports that under matched GB200 configurations, decoding throughput differs from DeepInfer's published results by less than 6%.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

Evaluate the entire inference stack, not just open-source model benchmarks

Ryan Hanrui Wang, Senior Vice President of Applied AI at Nebius, points out that public benchmarks only reflect a model's best-case performance and cannot show whether agents can accurately complete tasks in production environments; therefore, the entire inference stack should be evaluated using internal benchmarks and harnesses. Nebius acquired inference optimization startup Eigen AI for approximately $643 million and provides inference services through its Token Factory platform. He recommends using a large model as a reference to scan multiple models and find the scale sweet spot; for example, 10B parameters may already be sufficient for customer service tasks, while 100B or even 1T parameters bring only marginal accuracy gains.

English translation is being prepared.

English translation is being prepared.

DeepSeek open-sources inference framework DeepInfer and codec components under Apache 2.0

DeepSeek has open-sourced, under the Apache 2.0 license, the inference framework DeepInfer that powers its online services and the API-identical codec component deepseek-recipe, and the company says inference code for future new models will also continue to be merged into this repository. The framework includes high-performance operators, distributed inference, speculative decoding, SWA Bounded Replay, and structured output, but the README shows that this release only includes the offline inference engine; the KV cache system, prefill-decode disaggregated deployment, and multi-replica load balancing are not included. On hardware, it currently supports only NVIDIA's Blackwell architecture SM100 and Huawei Ascend 950, and the only supported model is V4.1-Flash.

DeepSeek open-sources inference framework DeepInfer and token codec component deepseek-recipe

DeepSeek has open-sourced its internal large-scale cloud service inference framework DeepInfer and the production-grade token codec component deepseek-recipe. The former supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and can run on NVIDIA and Huawei Ascend platforms. DeepSeek says the high-throughput, low-latency inference service of DeepSeek V4.1 Flash is based on this framework, with a single cluster serving 155 trillion Tokens/day in throughput, at 38 US dollars per card per hour calculated at off-peak prices. The processing logic of deepseek-recipe is fully aligned with the official DeepSeek API. The team also explained that the KVCache management and load balancing components are deeply coupled with internal infrastructure, so the relevant code has not yet been fully open-sourced.

DeepSeek open-sources DeepInfer, the inference engine behind its API, with support for both Nvidia and Huawei Ascend

DeepSeek has open-sourced DeepInfer, its in-house inference framework used to serve its models in production. Released under the Apache 2.0 license, it lets developers run DeepSeek-V4.1-Flash on their own infrastructure, with support for subsequent models planned. The framework includes optimized compute kernels and communication libraries such as FlashMLA, DeepGEMM, and DeepEP, supports speculative decoding, asynchronous scheduling, and structured output, and also open-sources the deepseek-recipe library (available in Rust and Python versions) for handling request and response transformations. On the hardware side, it supports Nvidia SM100 (Blackwell) architecture and the Huawei Ascend 950 NPU series from day one; older architectures such as Hopper are not listed. Batch-invariant inference and deterministic sampling are available only on Nvidia hardware, and the published performance data is also based on Nvidia GPUs. The current version supports offline inference but lacks complete production serving components such as KV cache management, cross-machine inference stage separation, and replica load balancing.

Zyphra Inference Wave 2: Accelerating MoE and Long-Context Inference Serving on MI355X

Zyphra has released ZOE, its second-generation inference optimization, which on AMD Instinct MI355X reduces GLM-5.3's time to first token (TTFT) at 512K context from 178.8 seconds to 64.6 seconds (2.8x), and from 42.3 seconds to 11.8 seconds (3.6x) at 90% prefix cache hit rate. Its CSA top-K selection HIP kernel raises DeepSeek-V4-Pro's single-user decode throughput at 1M context from 43.5 to 61.1 tokens/s (1.40x). AMD MXFP4 quantization of GLM-5.3 shrinks the model from 756 GB to 410 GB, with benchmark scores such as AIME25 within 3 points of FP8.

English translation is being prepared.