DeepSeek open-sources inference framework DeepInfer and token codec component

I didn't expect this to be open-sourced too.

On the afternoon of October 20, DeepSeek open-sourced its internal inference framework DeepInfer, along with the accompanying token encoding/decoding component deepseek-recipe.

DeepInfer is currently the foundation supporting DeepSeek's large-scale inference services. It was released this time under the Apache 2.0 license and currently supports DeepSeek-V4.1-Flash.

This release quickly drew attention. Yesterday everyone was discussing how DeepSeek now lets anyone quickly deploy their own AI models; today, on second thought, wait—how is large-model inference this profitable?

DeepSeek disclosed that, based on this system, a single cluster serves a daily throughput of 155 trillion tokens, and at API off-peak prices, overall revenue reaches $38 per card per hour.

What does that mean? Someone calculated a gross margin of 82%: assuming the API you sell has high enough utilization and you rent GPUs at $7 per B300 per hour, there is still considerable margin left after deducting only the rental cost.

DeepInfer adds important data to today's discussion of AI service pricing, letting us know for the first time the engineering capability required to run models efficiently.

For ordinary users, calling a large model is just sending a request and waiting for an answer. On the server side, however, a large number of requests must be arranged to share chips, read model parameters, mobilize computing resources distributed across different devices, and control queuing and generation latency.

You may notice that the models are the same DeepSeek model, but the official API and third-party deployments do not feel quite the same. This is because, for the same model, different inference implementations can differ in how many users each card can serve simultaneously and how long users need to wait. Inference frameworks solve exactly this kind of problem: they aim to improve inference efficiency and optimize both the effective throughput per unit of compute and the response speed users experience.

In this respect, no one should be able to do better than DeepSeek itself.

DeepInfer is mainly aimed at large-scale deployment, supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, integrates capabilities such as optimized operators, distributed parallelism, speculative decoding, and asynchronous scheduling, and also supports output constrained by rules such as JSON Schema. DeepInfer was initially developed mainly based on vLLM, and also drew on SGLang and TensorRT-LLM.

Meanwhile, deepseek-recipe is responsible for converting API requests in different formats into the prompts and token sequences used by the model, and then restoring the output into responses in the corresponding format. It supports interface formats such as Chat Completions, Responses, and Messages, covering processing of thinking content, tool calls, and streaming output.

This is of great significance for third-party deployments: developers can reuse the official request processing and encoding/decoding logic, reducing the chance of behavioral differences when the same application is moved to another server. Whether a model can be smoothly integrated into existing Agent workflows often depends on these details as well.

The currently open-sourced repository provides offline inference and the core capabilities that support large-scale services. Of course, a complete online service still requires supporting components such as a KV Cache system, vision encoding and separate Prefill/Decode deployment, and cross-replica load balancing. Third parties have obtained an important engine, but system integration is still required for deployment.

The common goal of these deployment efforts is to make each card do less redundant computation and serve more requests.

Judging from the official deployment materials, DeepSeek's optimization approach includes organizing resources by inference stage, reusing context computation results, and increasing the effective output of each decode.

When a large model processes a request, it first processes the input context (that is, Prefill), and then gradually generates an answer (that is, Decode). Deploying the two stages separately allows resources to be arranged separately and prevents different loads from crowding together. This helps optimize both the waiting time for the first token and the subsequent generation speed.

Caching is especially suitable for long conversations and Agents. After an agent calls a tool, the next round of requests often again carries the preceding system prompt, tool descriptions, and history. If the intermediate computation results of the existing context can be reused, the overhead of repeatedly processing the same content can be reduced.

One notable point is the retention time of the KV Cache. The official disclosure says Agent traffic is retained for 3 days and chat traffic for 7 days. These caches are placed in an independent 3FS storage cluster, and every Prefill machine can read them. By combining multi-day retention, shared storage, and automatic matching, DeepSeek lets long contexts that have already been computed continue to play a role in more subsequent requests.

Others, including Anthropic, OpenAI, and Google, mostly retain caches for between 5 minutes and 24 hours.

Another example is DSpark speculative decoding: candidate tokens are generated first and then handed to the main model for verification. DeepSeek further arranges verification according to the expected acceptance rate of candidates, reducing computation spent on low-yield candidates. The official deployment Q&A estimates that, near the listed production operating point, dynamic-length verification improves throughput by about 4-7% compared with fixed verification of 5 candidate tokens.

Cumulatively, these optimizations change the amount of service each graphics card can deliver. Its prices are low, but it really is not losing money; as long as compute is not idle, and the more compute cards it buys, the more it earns.

Around the discussion of token costs, according to the current official price list, the APIs for DeepSeek-V4.1-Flash and V4.1 Pro are as follows.

Comparing service revenue and cost requires considering the composition of input, output, and cache hits at the same time. After all, the price of cache-hit input is only one-fiftieth of that of cache misses. Fortunately, DeepSeek itself gave the figure of $38.

Apart from the $7 for renting GPUs, even if other service expenses are added, there is still a considerable gap between the API price and the cost. This shows that efficient inference can potentially achieve both low pricing and substantial profit space—no wonder cloud service providers are all borrowing money to build data centers. On Hacker News, people's discussions are divided: some calculated a higher profit margin based on lower GPU costs, while others stressed that the investment over the entire lifecycle of the model must be calculated.

However, the earlier claim of "recovering the entire equipment cost within 10 months" now has some basis.

On the other hand, The Information disclosed last month that DeepSeek's annualized revenue reached about $1 billion, roughly corresponding to $2.74 million per day. If compared with $38 per card per hour, that is exactly equivalent to 3,000 cards.

DeepInfer provides a code starting point for optimized deployment. For most users, more thorough inference competition is expected to bring lower calling costs and a better response experience.

Who knows whether in the future everyone will "stop delaying Uncle Liang's AGI training," or whether, once V4.1 Pro is launched, DeepSeek's servers will be overwhelmed again?

References:

This article is from the WeChat public account "Machine Heart" (ID: almosthuman2014), author: Machine Heart, which follows DeepSeek, editor: Zenan, published by 36Kr with authorization.