DeepSeek open-sources inference framework DeepInfer

36Kr · AI

SemiAnalysis's hands-on testing claims Anthropic is running an A/B gray test on Claude subscription quotas

SemiAnalysis ran an Opus 5.5 agent workload on two identical $200 Max 20x accounts and found that one account's 5-hour window quota was 2.076 billion tokens, 19% higher than the other account's 1.74 billion tokens, but the weekly total quota fell from 6.579 billion tokens to 5.266 billion tokens, a 20% shrinkage. Claude Code technical staff member Lydia Hallie said the quotas were not adjusted and suggested users run the feedback command to submit their ID for investigation; core developer Boris Cherny said the team is urgently investigating. The article also mentions that OpenAI previously halved the quota of its $200 Pro plan and introduced a $500 tier, as well as DeepSeek's open-source inference framework DeepInfer and its deployment practice of 155 trillion tokens per day of throughput on a single cluster.

Phoenix Technology

StartLux open-sources decision model inference engine Prism, up to 5.3x faster inference on a single card

StartLux Origin Star open-sources its self-developed inference engine Prism, claiming it is the first inference engine built specifically for decision models and open-sourced. On a single H200, Prism processes a single short question 2.8x faster than the original vLLM 0.31 solution, and asking three questions in a row about the same material is up to 5.3x faster. The team says it achieves the speedup through four optimizations: narrowing the output layer computation range, forest-style computation, int8 integer arithmetic, and mixed scheduling; in a 231-question cross-engine comparison, the average per-question latency is 41 milliseconds.

Sina Tech

Zhou Hongyi on DeepSeek Open-Sourcing DeepInfer: In the AI Era, Enterprises Must Have Their Own "Sovereign AI"

Zhou Hongyi stated regarding DeepSeek's full open-sourcing of its internal AI inference framework DeepInfer that this reflects technological confidence and open-source inclusiveness, and is even more about technological sovereignty in the AI era. DeepInfer enables CPUs and GPUs to work simultaneously through optimized scheduling, reducing waiting and redundant computation. 360 has built a sovereign AI operating system covering computing power scheduling, sovereign large models, enterprise AI knowledge bases, agent factories, Agent Teams, and security governance, which has already been deployed in some government and enterprise units.

The Paper

Zhou Hongyi on DeepSeek's open source: technical confidence, the AI era needs "sovereign AI"

Zhou Hongyi, commenting on DeepSeek's full open-sourcing of its internal AI inference framework DeepInfer, said this reflects technical confidence and open-source inclusiveness, and is even more about technological sovereignty in the AI era. He revealed that 360 has built a sovereign AI operating system covering compute scheduling, sovereign large models, enterprise AI knowledge bases, agent factories, Agent Teams and security governance, and that it has been deployed in some government and enterprise units. A McKinsey 2026 report estimates that by 2030, 30% to 40% of global AI spending may be affected by sovereignty requirements, corresponding to a market size of about $500 billion to $600 billion.

Caixin

DeepSeek open-sources inference framework DeepInfer, compatible with NVIDIA GPUs and Huawei Ascend 950

On October 20, DeepSeek announced the open-sourcing of its internal large-scale cloud service inference framework DeepInfer, aiming to improve model inference efficiency and reduce deployment costs, compatible with NVIDIA GPUs and Huawei Ascend 950 chips. The framework has been used in the online service of the DeepSeek V4.1 Flash model, with a single cluster processing an average of 155 trillion Tokens per day, and revenue of about 38 US dollars per GPU hour calculated at off-peak electricity prices. This issue of the GPT Weekly also mentions news such as Anthropic launching the personal AI assistant Claude Duet and MiniMax breaking its IPO price in the Hong Kong stock market.

Wallstreetcn

Goldman Sachs: The AI Capex Retreat Is Equivalent to an Alternative Form of Quantitative Easing for the Bond Market

In a morning note, Goldman Sachs' Rich Privorotsky argued that the scale of bond issuance by hyperscale cloud computing companies is now comparable to that of US Treasuries, and that a contraction in AI capital expenditure would reduce bond supply pressure, with an effect akin to quantitative easing. Earnings reports from several tech giants have raised doubts about the sustainability of AI capital expenditure; yields on 2-year to 10-year US Treasuries fell by more than 15 basis points in a single day, the 10-year yield dropped below 5%, and the Philadelphia Semiconductor Index fell more than 7%. Tesla cut its full-year capital expenditure guidance from more than $25 billion to about $20 billion, Meta raised the lower bound of its full-year capital expenditure guidance to $135 billion, and DeepSeek's open-source reasoning model DeepInfer claims it can substantially improve inference efficiency on the basis of existing computing power.

cnBeta

DeepSeek Open-Sources Internal Inference Framework DeepInfer

English text is being prepared.

Phoenix Technology

The DeepSeek WeChat official account cover mistakenly wrote "inclusive" as "HP", having previously open-sourced the inference framework DeepInfer

The cover of the DeepSeek WeChat official account article "DeepSeek Open-Sources Its Inference Framework: Toward a Truly Inclusive Future" mistakenly typed "inclusive" as "HP", colliding with the computer brand HP (Hewlett-Packard); the cover has now been corrected back to "Toward a Truly Inclusive Future". The article announced that DeepSeek will fully open-source its internal core inference framework DeepInfer, making public to the community the underlying inference engine and deployment solution that support large-scale online services, helping developers and enterprises improve throughput, reduce latency, and build private or public cloud AI services.

WallstreetcnSelected

DeepSeek open-sources production-grade inference framework DeepInfer

On October 20, DeepSeek announced the open-sourcing of its internal production-grade inference framework DeepInfer, which integrates technologies such as distributed inference and speculative decoding and is already compatible with NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Bloomberg noted that inference technology is usually regarded by AI companies as a core competitive advantage and is rarely made public; this open-sourcing gives outsiders a look at how DeepSeek keeps operating costs down. The documentation also states that building a complete service still requires a KV cache system, Prefill/Decode disaggregation, and load balancing across data-parallel replicas, which are not fully included in this open-source project.

36Kr · AISelected

DeepSeek open-sources inference framework DeepInfer and token codec component

On the afternoon of October 20, DeepSeek open-sourced its internal inference framework DeepInfer and the accompanying token codec component deepseek-recipe under the Apache 2.0 license, currently supporting DeepSeek-V4.1-Flash. According to official disclosures, a single cluster serves a daily throughput of 155 trillion tokens; at the API off-peak price, this amounts to $38 revenue per card per hour, and some calculated a gross margin of 82% based on a rental cost of $7 per hour per B300. DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, integrates capabilities such as speculative decoding and asynchronous scheduling, and was initially developed mainly based on vLLM; the company says KV Cache is retained for 3 days for Agent traffic and 7 days for chat traffic.

t3n

DeepSeek open-sources the Deepinfer inference framework, lowering the cost of self-hosted model deployment

DeepSeek has open-sourced its internal inference framework Deepinfer on GitHub, with the goal of reducing the compute and energy costs of deploying models on your own. During the inference stage, the framework distributes load more evenly across available processor cores, uses speculative decoding to speed up responses, and improves the structure of results before output; it currently supports only DeepSeek-V4.1-Flash, with more of its own models to be opened up later. On the hardware side, it requires Nvidia GPUs with the SM100 architecture or Huawei Ascend 950 NPUs, so it is not aimed at individual users.

ai_business

English translation is being prepared.

kr36_ai

English translation is being prepared.

X.PINSelected

DeepSeek Open-Sources Its Internal Inference Framework DeepInfer

DeepSeek has open-sourced its internal inference framework DeepInfer, which it says powers the high-throughput, low-latency inference of DeepSeek-V4.1-Flash. The framework supports both Nvidia and Huawei Ascend hardware, and offers capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output. DeepSeek also released deepseek-recipe, a production-grade tokenization component fully aligned with the official API.

cnbeta

English translation is being prepared.

Zhidx

OpenAI API adds a 12-hour cache retention option; caches for logged-in Codex and Work users are retained permanently

The API service for OpenAI models will offer a 12-hour cache retention option, and caches for logged-in Codex and Work users will be retained permanently. On the API side, this option applies to GPT-5.6 and later models, with a default retention of 30 minutes; when 12 hours is selected, the cache write price rises from 1.25 times the input token price to 2.5 times, while the cache read price remains unchanged. OpenAI's documentation states that the 12 hours specifies the minimum lifetime corresponding to eligibility for cache-read billing; it does not guarantee that the stored KV tensors can be reused on every request, nor does it guarantee reduced latency.

Latent SpaceSelected

DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day in a single cluster

DeepSeek has open-sourced DeepInfer, its self-developed inference framework that supports DeepSeek-V4.1-Flash, and published production data: a single cluster processes 155 trillion tokens per day, 96% input cache hit rate, 400ms first-token latency, and 230 tokens/s generation speed. The framework is customized based on vLLM and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, with target platforms NVIDIA SM100 and Huawei Ascend 950. DeepSeek also states that components such as KV cache management and load balancing are not yet fully open-sourced, while the vLLM team says that under a matching GB200 configuration, decoding throughput is within 6% of the results published by DeepInfer.

Latent Space

DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day on a single cluster

DeepSeek has open-sourced DeepInfer, the production inference framework supporting V4.1 Flash, and published production data including 155 trillion tokens processed per day on a single cluster, a 96% input cache hit rate, an average 400ms time to first token, and 230 tokens/s generation. The framework is customized based on vLLM and uses the Apache 2.0 license, supports distributed inference, speculative decoding, SWA Bounded Replay and structured output, and targets NVIDIA SM100 and Huawei Ascend 950. DeepSeek says components such as KV cache management and load balancing are not fully open-sourced because they are coupled with internal infrastructure, while the vLLM team reports that under matched GB200 configurations, decoding throughput differs from DeepInfer's published results by less than 6%.

shao_meng

English translation is being prepared.

Electronic Technology Application

DeepSeek Fully Open-Sources Its Internal Inference Framework DeepInfer

DeepSeek announced that it is fully open-sourcing its internal inference framework DeepInfer, and is simultaneously open-sourcing the production-grade token codec component deepseek-recipe, whose processing logic is fully aligned with the official API. DeepInfer integrates high-performance operators and an inference engine, supports capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and is compatible with NVIDIA and Huawei Ascend platforms. DeepSeek stated that it has already used this framework to deliver high-throughput, low-latency large-scale inference services for DeepSeek V4.1 Flash, and that inference code for future new models and subsequent optimizations will continue to be updated.

caixin_tech

English translation is being prepared.

siliconangle

English translation is being prepared.

the_decoder

English translation is being prepared.

interconnects

English translation is being prepared.

bjnews_ai

English translation is being prepared.

The New StackSelected

DeepSeek open-sources DeepInfer, the inference engine behind its API, with support for both Nvidia and Huawei Ascend

DeepSeek has open-sourced DeepInfer, its in-house inference framework used to serve its models in production. Released under the Apache 2.0 license, it lets developers run DeepSeek-V4.1-Flash on their own infrastructure, with support for subsequent models planned. The framework includes optimized compute kernels and communication libraries such as FlashMLA, DeepGEMM, and DeepEP, supports speculative decoding, asynchronous scheduling, and structured output, and also open-sources the deepseek-recipe library (available in Rust and Python versions) for handling request and response transformations. On the hardware side, it supports Nvidia SM100 (Blackwell) architecture and the Huawei Ascend 950 NPU series from day one; older architectures such as Hopper are not listed. Batch-invariant inference and deterministic sampling are available only on Nvidia hardware, and the published performance data is also based on Nvidia GPUs. The current version supports offline inference but lacks complete production serving components such as KV cache management, cross-machine inference stage separation, and replica load balancing.

marktechpost

English translation is being prepared.

36Kr · AISelected

DeepSeek open-sources inference framework DeepInfer and Token codec component deepseek-recipe

DeepSeek open-sources its self-developed inference framework DeepInfer (Apache 2.0) and the production-grade Token codec component deepseek-recipe (MIT), supporting DeepSeek V4.1 Flash in serving 155 trillion Tokens per day on a single cluster. Official online statistics show an average of 77K Tokens input per request, a time to first token of about 400 milliseconds, 230 Tokens output per second during the generation phase, and an average of 96% of input hitting the cache. DeepInfer currently supports V4.1 Flash; the open-source version provides only offline inference, and the KV cache management and load balancing code has not yet been fully open-sourced because it is tied to internal infrastructure.

Tianyi Cui

DeepSeek open-sources the inference framework DeepInfer and components such as deepseek-recipe

DeepSeek has open-sourced its in-house inference framework DeepInfer, and simultaneously released key components for deploying DeepSeek models, including deepseek-recipe. DeepInfer already supports large-scale serving of DeepSeek-V4.1-Flash in production, with a focus on high throughput and low latency. DeepSeek said that publishing its production stack is intended to help the community lower compute costs, reduce energy consumption, and enable more organizations to deploy DeepSeek models.

rohan_paul

English translation is being prepared.

thom_wolf

English translation is being prepared.

Nathan Lambert

DeepSeek open-sources its inference framework DeepInfer

DeepSeek announced that it is fully open-sourcing its self-developed inference framework DeepInfer, which is used to support large-scale, high-throughput, low-latency operation of DeepSeek-V4.1-Flash. The company said it is sharing its production stack to help the community lower compute costs and energy consumption and make AI more affordable. Nathan Lambert commented that this is a distinctive way to gain model adoption and is good for the community, and noted that the timing is delicate because the government has just banned DeepSeek in federal agencies.

ayi_ai_notes

English translation is being prepared.

thepaper_aiSelected

English translation is being prepared.

ZhidongxiSelected

DeepSeek open-sources inference framework DeepInfer and token codec component deepseek-recipe

DeepSeek has open-sourced its internal large-scale cloud service inference framework DeepInfer and the production-grade token codec component deepseek-recipe. The former supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and can run on NVIDIA and Huawei Ascend platforms. DeepSeek says the high-throughput, low-latency inference service of DeepSeek V4.1 Flash is based on this framework, with a single cluster serving 155 trillion Tokens/day in throughput, at 38 US dollars per card per hour calculated at off-peak prices. The processing logic of deepseek-recipe is fully aligned with the official DeepSeek API. The team also explained that the KVCache management and load balancing components are deeply coupled with internal infrastructure, so the relevant code has not yet been fully open-sourced.

ithome_ai

English translation is being prepared.

Ma Dongxi NLP

DeepSeek Open-Sources Large-Scale Inference Framework DeepInfer Under Apache-2.0 License

DeepSeek has open-sourced its high-performance large-scale inference framework DeepInfer under the Apache-2.0 license. The framework supports optimized kernels, model and sampler runtime, parallelism strategies such as Attention DP and balanced CP, MoE EP plus EPLB, as well as DSpark and confidence-scheduled verification, SWA bounded replay, asynchronous scheduling with a zero-overhead batch scheduler, batch-invariant inference, deterministic sampling, and structured output. Ma Dongxi NLP reposted the news and said "Respect."

Shao MengSelected

DeepSeek Open-Sources High-Performance Inference Framework DeepInfer

DeepSeek has open-sourced DeepInfer, the high-performance inference framework supporting its online services, under the Apache 2.0 license. This release focuses on offline inference, is built primarily on vLLM, and draws on SGLang and TensorRT-LLM. It integrates previously separately open-sourced kernels such as FlashMLA, DeepGEMM, DeepEP, and EPLB into a complete engine, filling in system layers such as scheduling, parallelism, and speculative decoding, while supporting NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The original article also breaks down designs including DSpark speculative decoding, YoCo SWA bounded replay, asynchronous scheduling, batch-invariant deterministic inference, and regex/JSON schema/EBNF structured output.

deepseek_ai

English translation is being prepared.

Tianyi CuiSelected

DeepSeek open-sources the large-scale inference framework DeepInfer under the Apache-2.0 license

DeepSeek has open-sourced the high-performance large-scale inference framework DeepInfer under the Apache-2.0 license. The framework supports optimized kernels, model and sampler runtimes; on parallelism it provides Attention DP, balanced CP, and MoE EP + EPLB, and includes DSpark and confidence scheduling validation, SWA bounded replay, asynchronous scheduling of a zero-overhead batch scheduler, batch-invariant inference, deterministic sampling, and structured output.