SiliconFlow's Tang Anbo: Using the Token Productivity Formula to Help Central SOEs Land Large Models at Scale

At the 2026 Autumn Central SOE and State-Owned Enterprise CIO and Digital Technology Company Executive Summit (the 19th) hosted by enterprise network D1net, SiliconFlow Solutions Director Tang Anbo attended the summit and gave a keynote sharing: "From 'Compute' to 'Token Productivity': Inference Infrastructure and ROI Practices for Large-Scale Deployment of Large Models in Central SOEs and State-Owned Enterprises". The following are highlights of the sharing content.

Currently, AI applications are in the early stage of explosive growth, and Token consumption is growing exponentially. Behind this, model architectures continue to evolve, with rapid version update cadence; at the same time, compute architectures are also continuously evolving. Between models and compute is a mutually driving iterative process. For enterprises, on the one hand they must adapt to these changes, and on the other hand they must provide diversified model services for their own business scenarios, while also keeping costs within a reasonable range.

What SiliconFlow does is bridge the conversion layer from compute to Token: uniformly manage compute; through a self-developed inference engine, make models run faster and better. We build Token factories on the public cloud, and also help central SOEs and state-owned enterprises build their own Token factories internally.

In short, through systematic inference engineering, we achieve multiple adaptations between models and compute. Based on this, SiliconFlow proposes the core formula of Token productivity: Token productivity = capacity × utilization × delivery quality. Among them, capacity is jointly determined by chip hardware capability and model optimization level; utilization is achieved through traffic governance and intelligent compute scheduling; delivery quality includes model inference accuracy, task execution success rates such as tool calling, and the SLA service stability essential for production environments.

From unified management to model operations, truly raising compute utilization

From the perspective of business and traffic, we designed a four-layer decoupled architecture. It connects model operations, model maintenance, inference execution, and compute scheduling, covering every link in a model's full lifecycle from deployment and acceleration to service scheduling. The layered decoupled architecture design is also intended to adapt to the existing platform construction of central SOEs and state-owned enterprises and avoid duplicate construction; if an enterprise already has self-developed capabilities, it can integrate with existing systems without starting over, and can directly fill in the required capabilities in the form of "capability composition".

Model gateway: not only an entry point, but also a model operations platform.

This layer directly faces users of model services, such as local branches and subordinate groups. All large model calls complete basic work such as metering, billing, and authentication through this layer. More importantly, it also provides rich scheduling and routing policies: for example, according to the response time requirements of different business scenarios, requests can be routed to the corresponding model services, so that businesses do not affect each other and all parties obtain satisfactory SLAs; it also supports flexible scheduling according to different objectives such as performance priority and cost priority.

Model service platform: deliver best practices for deployment and maintenance to enterprises.

This layer mainly manages capabilities such as model onboarding and offboarding and model repositories from an operations perspective. We put model deployment best practices into it, so that enterprises can launch models with one click instead of redoing adaptation each time. We have already adapted nearly a hundred models, and continue to adapt new mainstream models, all with the goal of enabling business teams to use them out of the box.

High-performance inference framework: optimization iterations for specific models and hardware.

We carry out in-depth cooperation with different compute vendors, and around specific models and compute, self-develop operator libraries and communication libraries to release model performance on the corresponding hardware as much as possible. For example, in cooperation with Huawei Cloud, we launched the industry's first DeepSeek R1/V3 Token service based on domestic chips, and jointly developed and optimized an inference solution for the CloudMatrix super node cluster, achieving single-card Decode throughput exceeding 1,920 tokens/s on DeepSeek-R1; world-first commercial service deployment of the Kimi-K2 large model based on the MetaX XiYun C550 cluster; deployed DeepSeek-V3 on Moore Threads MTT S5000 chips, achieving single-card Prefill throughput exceeding 4000 tokens/s and Decode throughput exceeding 1000 tokens/s.

At the same time, our inference technology is also continuously evolving: as the context of model processing tasks continues to grow and concurrency continues to increase, the inference framework needs continuous optimization, such as operator optimization, communication optimization, model quantization, etc., while combining PD separation and KV Cache hierarchical offloading and other means to comprehensively improve the overall performance and throughput of model inference.

Compute scheduling: elastic management of compute resources.

Elastic and flexible compute scheduling is a key element in reducing inference costs and improving compute utilization efficiency. But compute planning often faces a dilemma: if estimated too conservatively, it may soon reach full load or even overload; if configured entirely according to daytime peaks, it will cause resource waste in other periods. Therefore, both elastic scaling policies and operational measures are needed.

For example, when central SOEs and state-owned enterprises provide model services to their branches, they can adopt time-of-use pricing in internal settlement: regular pricing during the day, discounts at night, guiding some businesses to run at night. Different businesses may use different models, which requires the platform to dynamically adjust model instances based on multi-dimensional indicators such as request queues and performance.

Central SOE case: shifting from "renting machines" to "calling model services"

We have served a central SOE group that originally had about 50 machines and provided services to subordinate branches. The early service method was very direct: branches applied for machines and deployed the models they wanted to use. To put it plainly, they were "renting machines".

As a result, forty or fifty model instances were opened, but overall compute utilization was very low. Some units, after occupying machines, deployed only one model; some used them during the day but not at night; others used them only a few days per week. The machines were occupied, but did not continuously generate effective load.

We worked closely with the customer to restructure around the above situation: unified models, unified compute. We consolidated duplicate models down to about ten; we changed from allocating resources per machine to a Serverless model service, where everyone calls a unified API and no longer cares where a model is deployed or which machine it occupies.

This way, the same roughly 50 machines can do more. For example, after using some machines to host existing smaller models, we can free up several machines to deploy new large-parameter models, such as GLM-5.2, DeepSeek-V4-Flash-0731, and others. In the past, such models could not be run within the group. But in fact, new large-parameter models are very important for business exploration, because the upper limit of model intelligence also determines the upper limit of application innovation. Through model consolidation and intensive compute management, we deployed these models, and combined with performance optimization, made more scenarios go from "unusable" to "usable," and from "explorable" to "scalable."

Previously, the biggest concern on the business side was: doing exploration, building an Agent, is no problem, but can it be launched at scale? Can it handle more calls? Everyone had questions in their minds. After the new models went live, in tests from single concurrency to higher concurrency, output speed overall basically stayed above 30 tokens/s. Take programming as an example: exceeding 30TPS is the baseline requirement for a good user experience. In comparative tests of the same model, we achieved a 40%–100% performance improvement. Of course, this depends on different input and output lengths, and the specific gains differ.

Overall, first, model throughput increased; second, compute utilization increased. The two combined are ultimately reflected in the cost per million Tokens, an indicator many enterprises use to measure inference service efficiency. In this case, the cost per million Tokens for a Qwen model was originally over one hundred yuan, and after adjustment it dropped to under ten yuan. The cost reduction is significant, but the most critical thing is that it opened up more business scenarios that were originally difficult to implement.

Further improving ROI: optimizing applications and infrastructure together

As Agents scale up, we have also observed a problem: there is a certain disconnect between the application layer and the infrastructure layer. In the past, when building applications, even developers mainly responsible for the frontend usually had some understanding of database indexes and SQL optimization. Now, the layering between large-model applications and infrastructure is fairly pronounced, and the application side may not fully understand the underlying inference and caching mechanisms, which directly affects efficiency.

Take Coding as an example. We are exploring optimization in this area together with the customer. Software engineering in central state-owned enterprises is not about writing an App today and doing another demo tomorrow; it requires continuous iteration over many years. Code repositories are constantly updated, and committing code and running builds every day are part of a long-term engineering effort. How this code is constructed into the model's Context is very critical.

If the context is poorly constructed, every request fails to hit the cache, and inputs of hundreds of thousands of Tokens consume a large amount of compute. But if constructed well, about ninety percent of the content may hit the cache, and the efficiency gap becomes apparent. For example, without cache hits, the system may support 100 concurrent requests; after improving cache hit rates, it may support 200. This improvement does not add hardware; it reduces repeated computation. Therefore, Context construction itself needs to be optimized for inference.

Another direction is optimization combined with DevOps. Compute usage has peaks and valleys; during the day everyone is using compute, and inputs in Coding scenarios are very long. If all code is processed only when requests are initiated during the day and caches are generated then, it will occupy a large amount of compute during peak hours. So, can we, during the daily build process every night, use engineering means to have the model preprocess the code repository and pre-store the corresponding computation results in the KV Cache? That way, there is no need to recompute during the day. These approaches all have the potential to further improve compute utilization. But this capability also depends on joint exploration with the scenario owners.

Leiphone Leiphone