DeepSeek open-sources production-grade inference framework DeepInfer

As American artificial intelligence (AI) companies focus on expensive closed-source models, DeepSeek has opened up its technical "family assets" to the outside world, winning praise from overseas netizens.

On October 20, DeepSeek announced the open-sourcing of its internal production-grade inference framework, DeepInfer.

This framework is central to DeepSeek's ability to keep its large models closely trailing top US AI models in the face of US export restrictions on cutting-edge chips. The inference framework is responsible for managing how models use computing resources to handle user requests; its design affects latency, throughput, and the hardware needed to run the models, making it an important factor in AI operating costs.

US Bloomberg News pointed out on the 21st that inference technology is generally regarded by AI companies as a core competitive advantage and is rarely made public, whereas DeepSeek's disclosure of its inference framework this time gives the outside world a glimpse into how it keeps AI model performance high while driving down operating costs.

DeepInfer integrates multiple technologies such as distributed inference and speculative decoding, enabling AI workloads to be distributed across different hardware, thereby allowing models to generate more output in a shorter time and further tapping into chip performance. At present, the framework is compatible with NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

Many of these underlying technologies have already been widely adopted by the industry, but DeepSeek has specially optimized them for its own models so that these technologies can work together more efficiently.

US media said that for AI hosting service providers, lower inference costs mean greater profit margins and could further translate into lower AI service prices. For enterprise users, if they choose to run DeepSeek models on their own infrastructure, DeepInfer may also help them reduce computing costs.

Moreover, after DeepSeek open-sourced DeepInfer, other developers can also study its technical solutions and apply some of the methods to their own model inference systems.

Under the post in which DeepSeek announced the open-sourcing of DeepInfer, Unsloth AI, a California-based company focused on large language models and open-source AI optimization, local training, and inference, publicly thanked DeepSeek, saying that with the help of Dynamic GGUFs, it has enabled DeepSeek-V4.1-Flash-1017 to run locally on devices equipped with 128GB of memory.

The comment section also saw a large number of positive responses from overseas developers and AI practitioners. A social media blogger from India directly described DeepSeek as "Real OPEN AI" and paired it with a video poking fun at OpenAI CEO Sam Altman.

A software developer from Bangladesh marveled that DeepSeek may be one of the companies that has contributed the most to the open-source large language model ecosystem.

Of course, what DeepSeek released this time is not a complete inference technology stack. Its documentation mentions that building a complete service still requires features such as a KV cache system, Prefill/Decode disaggregation, and load balancing across data-parallel replicas, which are not fully included in this open-source project. However, most of the core disaggregation functions have already been implemented.

Thibault Jaigu, CEO of AI infrastructure startup Requesty, said that DeepInfer lowers the threshold for deploying DeepSeek models, but enterprises still need the corresponding computing power and technical capabilities. He believes that DeepSeek is in effect telling the outside world that "anyone can self-host our models, but there are still some tricks to note if you want to achieve efficient management."

In the eyes of many overseas netizens, what DeepSeek has done is not just release an open-source inference framework; it is making more and more technologies that originally belonged inside companies public, allowing more people to participate. This stands in sharp contrast to the US AI industry's growing emphasis on closed-source models, massive computing investment, and commercial moats.

Although OpenAI has "open" in its name, its core models have long followed a closed-source strategy; instead, it is China's DeepSeek that is practicing the word "open" through concrete actions.

"Do good deeds and let others share them; master skills and let others share them." As DeepSeek said, "We look forward to seeing, in the near future, the scope and quality of global AI services expand and improve as never before, becoming fairer, more transparent, and more trustworthy, thereby accelerating technological iteration and reducing energy waste across society."

Source of this article: Daju Finance. Risk disclosure and disclaimer

Markets carry risks, and investment requires caution. This article does not constitute personal investment advice and does not take into account the specific investment objectives, financial situation, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article are suitable for their particular circumstances. Investment based on this is at the user's own risk.