In the inference era, the computing power industry is changing its accounting books: Token cost and engineered delivery become the new yardsticks

There has been a very obvious change in the computing power industry this year: users are no longer dwelling on "how large the computing power scale is" or "how much chip performance has improved," but are instead repeatedly circling around a more specific word: Token. Yang Zhi, co-founder of Xin'aisuan, told us that since the beginning of this year, the questions their customers ask have completely changed. "Customers have started to focus more on how many tokens a model can output per second when running, and how much it costs to generate a token?"

The logic of the computing power business has changed

In Yang Zhi's view, today's intelligent computing industry is like the real estate industry twenty years ago. "Back then, real estate involved contractors doing customized work and work units allocating housing; you found a contractor to build the house and then you could move in. Many intelligent computing centers today are the same: a big company comes to rent the data center space, moves its equipment in and sets it up, and that's the end of it."

Chips are the bricks, card modification production and maintenance are prefabricated construction and property management, and investment and construction management are real estate development. "Companies like Xin'aisuan do not participate in IDC land acquisition and construction, but instead inject computing card assets into intelligent computing centers in places such as Zhengzhou and Hangzhou Xiaoshan," Yang Zhi said.

But what truly matured the real estate industry was not building towers, but the shift from turnkey projects to refined asset operations, and this is exactly the step the computing power industry is going through now. This shift has been further driven by the explosive demand for AI for Science (AI4S). Statistics from QYResearch show that the global AI for Science market was about US$4.538 billion in 2025 and is expected to reach US$26.23 billion by 2032, with a compound annual growth rate of 28.9%.

The demand for AI4S is also a reflection of the booming inference market. Liu Liehong, director of the National Data Administration, also disclosed at the 2026 Digital China Summit that inference data volume had reached 101.34 exabytes in 2025, surpassing training data volume for the first time, and that in the future the ratio of inference computing demand to training computing demand may reach 3:1 or even higher.

The CAICT "Token Economy Development Research Report (2026)" also points out that in 2026, inference computing power will account for 70.5% of the total AI computing load. Gartner's forecast is even more directly reflected in where the money flows: global inference spending will reach US$23.3 billion in 2026, surpassing training spending of US$19 billion for the first time and accounting for 55% of total AI IaaS spending. The "lead and supporting role" relationship between training and inference officially flipped this year.

This turning point on the demand side has already been transmitted directly to the procurement and acceptance stages. Yang Zhi admitted that in the past customers only looked at card models, quantities and peak FLOPS, but now they ask: for their own models, context lengths and concurrency requirements, what throughput can be achieved? Is response latency stable? How are monthly energy consumption and O&M costs controlled? How long does it take for the business to recover if something goes wrong? "FLOPS is still a basic indicator for equipment selection, but it is no longer the endpoint of procurement decisions and project acceptance. What customers buy is not a set of parameters, but a business capability that can run sustainably," Yang Zhi pointed out.

This also explains why new indicators such as tokens produced per watt and inference latency per unit cost have become popular in the industry. Chen Jian, chairman of Beijing Paratera, once said at the Zhongguancun Forum that the gap between the industry's best and worst token performance and cost can reach 100 times; the cost of one million tokens on DeepSeek's official website fell from about 130 yuan at the beginning of the year to below 3 yuan through system optimization, and even as low as 0.3 yuan in offline scenarios.

Even so, the "climax" of inference demand has not yet arrived. In this regard, Yang Zhi pointed out that the current first wave of demand comes from consumer-facing agent products. When an agent completes a full set of tasks, it consumes several times or even a hundred times the tokens of ordinary chat; after many products launched, token usage directly rose sixfold, and services urgently expanded capacity tenfold. And when large models in finance, healthcare, manufacturing, energy and other industries truly enter every sector, "when the door opens, it will be at least 100 times the current computing power demand," Yang Zhi judged.

What is truly difficult is not buying cards, but getting the system running

If the token economy has changed the accounting book of computing power, then engineering delivery capability is the key to whether this book can be balanced. He Mujun, president of Sugon Intelligent Computing, told us that nowadays when various foundation model vendors release new models, the supercomputing internet can basically achieve "zero-stack adaptation," because the computing compatibility of domestic chips is getting better and better, and foundation model vendors also basically train on the CUDA architecture. But being able to run and being easy to use are completely different things.

Take model context as an example: foundation models basically all support million-level (1M) context, but in actual deployment, 0-128K, 128K-512K and 512K-1M each suit different deployment and optimization schemes, and one must also consider how to deploy PD separation, as well as issues such as parallelism. "This workload is very large; we basically work backward from the model release, starting two to three months in advance," He Mujun pointed out.

A model "lighting up" is only the starting point; there is still a great deal of systems engineering between that and stable business operation. On this basis, Yang Zhi split delivery validation into four layers: the first layer is device health, including power supply, identification of cooling components, and basic stress testing; the second layer is cluster and network compatibility, including driver firmware, network links, and communication topology; the third layer is business load validation defined together with the customer, including models, concurrency, accuracy, and critical latency; the fourth layer is continuous operation and acceptance handover. "A device being able to power on does not mean the cluster can communicate stably, and a cluster passing tests does not mean the customer's models and processes can meet expectations," Yang Zhi emphasized.

As for the standard of being easy to use, in Yang Zhi's view, it can be evaluated from four dimensions: performance is predictable, with no major fluctuations under critical concurrency; operation is stable enough that long tasks do not drop or interrupt; faults can be localized, so customers do not have to guess repeatedly among modules, baseboards, networks, and software; and services can be restored, with clear response, spare parts, repair, and re-verification mechanisms.

On the software side, the room for optimization is likewise enormous. Yang Zhi divided inference optimization into three layers: the model layer, the engine layer, and the system layer, and the system layer—"getting a whole cluster of machines to work"—is precisely the most fiercely contested battlefield in the industry. The Agent KV cache bottleneck solved by DeepSeek together with Tsinghua brought online throughput close to doubling; public data released by Nvidia shows that on the same hardware platform, with software optimization alone, DeepSeek's token cost fell to as low as one-fifth of the original within a month; PD separation, FP4 expert parallelism, and a 20-fold increase in card-board throughput have all become verified facts.

This also makes heterogeneous computing an inevitable choice. In the training era, what mattered was ultra-large-scale clusters, high-speed interconnect, and a mature software ecosystem, none of which could be missing, and Nvidia was almost the sole dominant player; but the rules of the game on the inference track have completely changed. Inference competes on the cost, energy efficiency, and business throughput of every token. "Upper-layer businesses increasingly do not care what kind of card is underneath," Yang Zhi pointed out.

In the inference era, giants begin to stake out positions

The rise of inference has not only changed the original computing ledger, but is also rewriting the structure of the chip industry. Take industry giant Nvidia as an example. Last December, Nvidia announced a non-exclusive licensing agreement worth about $20 billion with inference chip company Groq, obtaining authorization for core technologies such as its LPU architecture. Founder Jonathan Ross and the core team joined Nvidia en masse, while Groq itself still operates independently. This transaction surpassed Nvidia's record of acquiring Mellanox for about $7 billion in 2019, becoming its largest deal ever.

What is noteworthy is the form of the transaction: it is not a traditional merger and acquisition, but "technology licensing + talent migration." The industry generally interprets it as a defensive acquisition. "When inference's requirements for latency, energy efficiency, and determinism go beyond the comfort zone of general-purpose GPUs, Nvidia chose to bring the most threatening non-GPU architecture into the fold," Yang Zhi further analyzed. "Nvidia took Groq, AMD got a route benchmarked against Google TPU, and everyone is desperately competing for ASIC inference chips. At a time when large models have gradually converged, the opportunity for ASICs is a blue ocean market worth trillions." In 2025, AMD's data center business revenue nearly doubled year on year, and a large number of ASIC inference chip startup teams emerged. Signals from overseas and domestically are all pointing in the same direction.

At the same time, the way of ecosystem collaboration is also evolving. The practice of the Supercomputing Internet provides another reference: through the "Ten-Thousand-Card Cluster Invitation Testing Plan," they let tens of thousands of users participate in testing applications of one hundred thousand cards, while in the actual customer base, the largest usage is instead inclusive demand within 10 nodes. A user with 80 cards today may have 800 cards or several thousand cards a few months later. He Mujun admitted that, from a financial perspective, making computing inclusive is not as cost-effective as selling to major customers all at once, but the Fudan brain-inspired team started with a few dozen cards and one hundred cards and has now reached the ten-thousand-card level. It is reported that the "Ten-Thousand-Card Co-Creator Incentive Plan" has already covered various participants such as scientific research, 985 universities, and pharmaceutical companies and hospitals lacking IT experience. Sugon is using a commercial approach to satisfy the different interest demands of all parties. "Not only must chips have different capabilities, but the entire workflow pipeline may need to handle different scientific tasks, which places further demands on system architecture and ecosystem building," He Mujun emphasized.

From the computing ledger to engineering capability, and then to ecosystem building, the changes that the inference era is bringing to the computing industry are deepening layer by layer. The yardstick for measuring computing power has been replaced by tokens, the focus of competition has fallen on delivery and operations, and the ways in which parties across the industry chain cooperate have also changed accordingly. As Yang Zhi said, the eve of the inference explosion often seems quiet, and that is precisely the moment for accumulating strength.