AI News

The money earned from Token exports can no longer keep up with soaring storage costs

TMTPost · AGIUpdated Nov 16 21:00

AI summary

Menlo Ventures' "2026 State of Enterprise AI Report" shows that among U.S. enterprise spending on open-source models, 89% goes to Chinese models, with Kimi accounting for 61%, DeepSeek 15%, and GLM 11%, yet open-source models account for only 4% of total enterprise model spending. The article points out that as long-session Agent tasks become the main force of overseas incremental traffic, context caching will continuously occupy VRAM and storage, while revenue is measured by Token and costs accumulate by GB; rising long-term storage contract quotes and compliance log retention further squeeze gross margins. The author believes the focus of competition is shifting from compute throughput to storage asset turnover and depreciation cadence.

Why read this

AI score 58

The article approaches from the cost structure, explaining why long-session Agent traffic creates a mismatch between Token-based billing and storage-occupancy-based payment.

Topics

Tags

Body · AI translation

By Silicon-Carbon Variables, Author: Wen Ning

A report titled "2026 State of Enterprise AI" released by the well-known Silicon Valley venture capital firm Menlo Ventures shows that of U.S. enterprises' spending on open-source models, 89% went to Chinese models. Among the top three, Kimi accounted for 61%, DeepSeek for 15%, and GLM for 11%.

The market is quite willing to accept a tidy narrative: relying on performance and pricing, domestic large models have captured overseas developers' budgets.

But the same report contains another set of figures: although open-source models have already taken a share of close to one-fifth (19%) of Token usage, open-source models actually account for only 4% of enterprises' total model spending, and the vast majority of enterprise AI budgets still flows to closed-source large models.

In other words, 89% is a share with very high concentration but limited volume.

Volume itself is not the biggest problem; the real change lies in its cost structure. You see, selling tokens is ostensibly about selling compute, but in essence it is about delivering state. And state does not attach to compute; it resides in memory and storage, and these are precisely the components where the current round of price-increase pressure is most concentrated.

The contradiction thus emerges: revenue is accounted for by token, but cost follows GB.

01. Traffic Is Running, Assets Are Sitting Idle

As the market keeps a close eye on the continuously updated Token call rankings of the major aggregation platforms and hotly debates the share battle of domestic large models in overseas open-source markets, most analyses still anchor their narrative on model capability and pricing advantages, assuming by default that this is ultimately a business of "compute for Token."

This understanding can still hold together in traditional dialogue scenarios, but in the Agent tasks that are currently driving overseas incremental traffic, the underlying logic of the load is beginning to diverge.

The resource model corresponding to traditional dialogue inference is intuitive enough, and it is also the foundation supporting the vast majority of current API pricing systems.

In this ideal framework: a user initiates a question, the inference cluster completes the computation and returns content; once a round of dialogue ends and the session is closed, the cache resources occupied by this request are released accordingly, and the hardware returns to idle.

Service providers can estimate costs based on total Token volume, and when there is still room in cluster utilization, the higher the throughput, the more opportunity there is to dilute unit overhead, so the logic of low-price volume running and trading scale for profit can hold together, and Token count and resource consumption are also assumed by default to be on similar growth trajectories.

Yet the main traffic force driving incremental growth in overseas aggregation channels is shifting from scattered human-machine dialogue to Agent tasks with automated pipelines.

This type of load has a clearly divergent resource rhythm from ordinary Q&A: at the task startup stage, some scenarios require injecting hundreds of thousands of Tokens' worth of front-loaded documents and batch business data; after that comes dozens of rounds of chained tool calls, continuously retrieving and reading external information; the entire session may be suspended for hours or even longer, and the same front-loaded context will be reused repeatedly.

More critically, absent actively closing the session, this portion of context cache will continue to occupy resources. The model continuously generates new Tokens, but merely appending to the cache does not automatically release this occupied block back to the resource pool.

As the share of long-session Agent traffic rises, the constraint point is also switching: what is being contested is not only inference speed; the upper limit of resident sessions the system can support simultaneously will become the new binding constraint.

The million-level context windows that vendors advertise externally are more of a product selling point aimed at customers; on the resource ledgers of the data center, it means that once a session uses up the window, each active concurrent path may need to reserve tens of GB of HBM space to carry the continuously online session state.

KV quantization, MLA, and various cache compression technologies on the market are all trying to lower the VRAM overhead per session and raise the concurrent scale the cluster can carry.

But in essence they only optimize unit cost and can hardly bypass that underlying constraint: as long as a session is in an active state ready to respond at any time, its core state data needs to reside in high-speed storage.

The expansion of this round of token traffic does not derive its core increment entirely from newly generated output text.

Trillion-level call volumes easily create an illusion: that growth comes from an endless stream of new content generation. In reality, a considerable portion of traffic growth corresponds to dynamic state assets that continuously pile up in the cluster, remain locked for long periods, and are hard to schedule and reclaim across tasks at low cost.

Platform statistics panels often record only the number of Tokens in and out, but in the service provider's operating ledger, the cost that truly keeps accumulating comes from those VRAM and high-speed memory resources that are occupied for long periods and hard to reallocate to other businesses at low cost.

The contradiction thus emerges: the industry's mainstream pricing framework is still born of the logic of the instantaneous-compute era, with revenue measured by input and output tokens. But the corresponding capital expenditure and high-speed VRAM occupancy costs will keep accumulating with the residence time of concurrent sessions.

Some Agents will hang suspended in operation for a whole day, producing very few tokens per unit time and generating limited revenue, yet VRAM remains continuously occupied. It inflates the call volume on the rankings, but revenue cannot keep up with the continuous consumption of resources, and the two growth curves that once tended to move in sync thus open up a gap.

English text is being prepared.

Previously, the competitive path on this track was simple and direct: compress the compute overhead per Token, use low prices to gain volume, and use scale to dilute hardware investment. When the load structure tilts toward Agents, sticking to the single competitive dimension of throughput and pricing may run into the trap of rising traffic but narrowing profit.

Rising call volumes easily send a signal that business is improving, but some service providers tend to overlook that the ever-accumulating session state assets are creating a continuous risk of eroding gross margin.

Beyond the bustling share rankings, the ability to manage state assets may be forming a new invisible dividing line.

02. Tokens going overseas, the invisible VRAM liability

When storage costs rise, the market easily simplifies it into a story of memory chip price increases. But viewed within the framework of inference cluster operations and capital expenditure, spot prices of components are only one of many variables.

What imposes a sustained rigid constraint on profit is often a whole chain of rules woven together from tied supply, asset depreciation, cross-border traffic and customer compliance.

Upstream memory chips are in a structurally tight-balance cycle, with long-term agreement quotes for server DRAM, enterprise-grade NAND for AI clusters and HBM continuing to rise, and overall inventories at original manufacturers remaining in a low range.

To secure stable supply, service providers building their own inference clusters can hardly rely solely on spot purchases to guarantee continuous expansion, and often need to sign quarterly or even semi-annual minimum-purchase long-term contracts in advance.

This also reshapes the nature of the investment: hardware procurement is no longer a variable input that can be flexibly adjusted with the business rhythm, but has become capital expenditure with rigid performance constraints.

Once such contracts are finalized, even if subsequent overseas traffic cannot fully fill cluster capacity, contract payments and asset depreciation must still be honored and accrued on schedule, and revenue fluctuations can hardly offset this rigid expenditure.

There is another cost that is easily obscured, coming from the compliance retention constraints of overseas enterprise customers. Many overseas enterprises, when purchasing inference services, require complete audit logs, call chains and even the original content of sessions to be traceable and auditable over the long term.

This persistent storage is independent of the KV cache temporarily generated during inference, and will occupy storage capacity stably over the long term.

Most standardized token procurement contracts do not list this storage service fee separately; the corresponding expense is spread into the unified Token quote, making it hard for outsiders to isolate and identify.

Even with optimization through KV cache compression, quantization schemes and iterative model versions, the physical service life of cluster hardware remains, but the effective business life cycle adapted to current Agent loads is being continuously compressed.

This forces service providers to lower the expected return period of assets, and the depreciation allocation per unit of token rises accordingly. With multiple constraints stacked, storage is no longer just a supporting cost of servers; under such loads, it is approaching the weight of compute power, and often becomes the bottleneck for cluster expansion.

Storage's erosion of profit is not a simple, general cost increase, but a continuous suppression of the cluster's return on assets along multiple independent paths. Such signals are not broken down and displayed in external price lists; most can only be observed in the operations and business reports in service providers' back offices.

First is the hit-rate trap of prefix caching. The underlying logic of prefix caching is to trade storage for compute reuse: cache frequently repeated context prefixes, and once hit, skip repeated computation to lower the compute overhead per Token.

But the cache itself carries write overhead, eviction scheduling logic and life-cycle constraints.

Once traffic becomes highly fragmented, tenants are dispersed and request randomness rises, the cache hit rate may quickly fall below the break-even point. A large amount of VRAM reserved for caching continues to occupy capital but can hardly deliver the expected compute savings, thereby lowering the return on assets of the entire cluster.

Second is the idle depreciation brought by peak reservation. To handle burst traffic, avoid sessions being forcibly cleared and prevent service stability from declining, the operations side often needs to reserve sufficient storage and VRAM redundancy.

Redundant resources are reserved for spikes and are mostly idle in normal times. Higher redundancy brings stronger stability, at the cost of lower asset turnover efficiency. Under cross-border tidal traffic, this depreciation cost remains rigid and is hard to fall in sync with off-season token revenue.

Finally, there is the silent resource lock-up caused by long sessions. Some Agent automation tasks have this characteristic: the token output rate per session is not high, but the session life cycle is extremely long, continuously occupying a segment of VRAM space.

Such loads do not quickly raise the call scale on platform throughput rankings, but they gradually squeeze VRAM resources and continuously consume the cluster's expansion headroom, possibly forcing service providers to start a new round of hardware procurement ahead of time.

The market is accustomed to judging the competitive landscape of domestic large models going overseas by token unit price and model capability, but under the new traffic structure dominated by Agents, the focus of competition is extending from a pure contest of compute power to a more refined management of asset efficiency.

While the industry's attention remains on token call rankings, storage asset turnover, the cache break-even line and the pace of depreciation are slowly becoming hidden yardsticks affecting the profit and loss of overseas business.

03. Will storage inflation reshape the large-model overseas business?

The pressure of rising storage costs draws a clear dividing line in the distribution chain: players holding hardware assets that carry long-resident sessions will bear the rigid pressure that is hardest to pass on externally.

Among them, some service providers control scheduling policies and caching rules and have the initiative in pricing to downstream, but their corresponding risk exposure is also the highest.

Many suppliers building their own clusters need to rely on quarterly or semi-annual minimum-purchase long-term agreements to obtain stable supply, and are highly likely to bear alone the asset idle costs brought by rising chip quotes and cross-time-zone traffic mismatch, as well as the impairment pressure from rapid hardware iteration.

In the stage of tight storage balance and continuous expansion, the incremental revenue brought by traffic growth is easily devoured quickly by continuously added capital expenditure.

Another mainstream approach is to enter revenue-sharing hosting partnerships with overseas cloud providers. From the perspective of the industry's books, this is often understood as the colloquial off-balance-sheet treatment of assets and liabilities: the model provider outputs model licensing, while the hardware investment and asset risks corresponding to memory, persistent storage, cross-time-zone operations and maintenance, and compliant log retention are mainly borne by the cloud provider.

But this is not equivalent to asset divestiture in the accounting sense, nor is it equity investment. If the contract sets minimum-usage or cost-linked clauses, part of the cost pressure may still be transmitted back.

In addition, there is another path: outputting only the model weights and relying entirely on an aggregation platform for distribution. Such suppliers basically do not hold inference hardware, so their direct exposure from storage and compute assets is the smallest, but their bargaining power is usually also the weakest.

The platform holds the authority to adjust routing allocation, traffic quotas, and caching policies, and suppliers mostly can only passively accept the traffic allocated to them. Even if load characteristics can be fine-tuned through versions and parameter configurations, it is still very difficult to independently intervene in the asset efficiency of the underlying clusters.

There is no absolutely perfect path. These choices are mostly trade-offs between traffic growth potential and storage asset exposure, while also taking into account other constraints such as compliance, customer ownership, and pricing initiative.

During an upward cycle in storage prices, off-balance-sheet asset treatment will become a more attractive choice for many players; only when upstream capacity is released and long-term agreement quotes fall back will the profit window for self-built clusters reopen.

The endless stream of KV quantization, sparse attention, CXL memory pooling, and tiered caching solutions in the industry easily creates an illusion: that relying on technical optimization can smooth away storage cost constraints.

But when it comes to the operating books, all kinds of engineering solutions have clear ceilings. They can raise asset utilization, but they cannot eradicate the rigid storage demand on the state side.

KV compression can reduce memory usage per session, often at the cost of additional compute overhead; sparse attention can cut computation, but cannot eliminate the need to keep session state resident; CXL memory pooling helps offload memory pressure, but retrofitting server hardware itself brings a new round of capital investment.

More critically, the benefits of optimization are highly tied to the traffic structure: Agent tasks with high context reuse can fully capture the cache dividend, while the returns from optimizing fragmented and one-off requests decay rapidly.

Technical means can raise the ceiling of asset efficiency, but it is very difficult to bypass the underlying constraint that session state must rely on hardware storage.

In the long run, disaggregated storage-compute architectures have the opportunity to deeply decouple computation from state storage, enable flexible migration of session state across nodes, and reshape the asset allocation logic of clusters.

But the reconstruction chain for the entire infrastructure is long, and it is very difficult to reverse the current cost landscape in the short term.

Compared with call-volume leaderboards that only reflect scale, another set of metrics better reflects the true quality of this business: the share of long-context requests, cache hit rate, average session lifetime, storage idle rate at overseas nodes, and quarterly quotes for long-term storage agreements.

A simple test is this: as call volume continues to rise, can gross profit per token be maintained or even improved?

If revenue continues to climb while gross profit stays flat or even declines over the long term, it often signals that incremental gains are being continuously eroded by storage depreciation.

Token going global is undergoing a quiet business-model shift: in the early days it was more easily categorized as a compute leasing business, with marginal costs expected to be continuously diluted as scale grows. But as long-session Agents gradually become the main traffic driver, the nature of this business is moving closer to online state hosting.

Memory and storage become core accumulated assets, marginal costs are hard to keep declining, and asset turnover directly determines the profit base.

Traffic can migrate at any time along the routing of aggregation platforms, but once hardware capital expenditure is made, the depreciation schedule will not adjust in sync with customer traffic. Matching highly liquid short-term orders with heavy assets depreciated over the long term naturally creates a cyclical mismatch between revenue and cost.

The market is always comparing total token call volume and share rankings, but from the perspective of the supply-side books, the more worthwhile question is: how many GB of storage must be locked in for each additional persistent session, and can this asset recover enough cash flow within the depreciation cycle?

Share is the vote cast by the demand side; the storage asset ledger is the core constraint that determines whether the large-model going-global business can run sustainably over the long term.

As tokens gradually become standardized commodities, what ultimately creates the profit gap may no longer be the compute used to generate text, but the storage assets that carry context memory.

Source

TMTPost · AGI