YC Paper Club Recap: Inference Compute Is 12x Training, Single-Card Compute Utilization in Decode Phase Only 5%

This session is YC's Paper Club. Four people working on inference took the stage in turn, and they were all talking about the same thing: the day model training finishes, the bill has only just started. The hardest numbers come from DeepSeek, and it is the only company in the world that has disclosed both its training and inference GPU-hours. Training V3 took 2.8 million H800 GPU-hours, and inference now burns 43,000 H800 GPU-hours per day. Over a year that comes to more than 15 million GPU-hours, more than five times training. YC partner Francois simply rounded it up to six times, then spread that over a two-year model lifespan to get twelve times. In other words, those 2.8 million GPU-hours of training account for only one-thirteenth of the model's total compute over its lifetime.

Training is a one-time payment; inference is renewed daily

He offered a supporting data point offhand: over the past decade, inference-related companies YC has invested in have produced $220 billion in market value. That number sounds abstract, but mapped onto the cost structure it is straightforward. Training is one-time; inference grows with the number of users, conversation turns, and context length. The longer a model lives and the more people use it, the more the bill tilts toward inference. So he said the habit in papers of casually writing train flops on the horizontal axis should change; aligning everything by train flops amounts to systematically underestimating the inference side. Where research funding goes gets skewed in exactly this way.

He added a blunt real-world point: in the inference business, many people do not even know what COGS is. It sounds like nothing more than a single model forward, but actually doing it means squeezing every cent out of the seams of the system. If you cannot squeeze it out, then the more you sell, the more you lose.

A GPU only runs at 5% during generation

The first set of numbers he gave was MFU, model compute utilization. Training and the prefill stage can reach 50%, but at decode, the stage where tokens pop out one character at a time, it drops to just 5%. When Nvidia sells you an H100, it says the card has 2 trillion FP8 operations; in practice you should celebrate if you get half. The reason is not complicated: every time decode generates a token, it has to move the entire model's weights from VRAM to the compute units. The bottleneck lands on memory bandwidth, while compute sits idle.

The data center numbers DeepSeek released show how wide this gap is. Prefill runs at 60,000 tokens per second per GPU, decode at 14,000 tokens per second per GPU, a gap of more than four times. If the input cache hits, prefill can surge to 1.5 million tokens per second. First token latency is 400 milliseconds, P99 one second. In terms of felt experience, the moment you press Enter is fast; watching it spit out one character at a time is the expensive part.

He used arithmetic intensity to explain this gap. Moving 4 bytes from VRAM on-chip, how many operations can you do. For images and diffusion models, one weight is reused repeatedly, so arithmetic intensity is high. LLMs are different: for every token generated, the entire model's weights pass through once, each used only once, so arithmetic intensity is pitifully low. No amount of spare compute can fill it; what you are waiting on is transfer speed.

A 96% cache hit rate, achieved by not deleting for three days

That 1.5 million token figure rests on a very crude trick. DeepSeek keeps users' KV cache for 72 hours, with PB-scale storage laid out there. If you repeatedly ask an agent about a codebase, the earlier context stays hot, and the second time you do not need to recompute. If you act like an elder at home and open a new conversation every time to ask about the weather, the cache is all wasted, and prefill runs from scratch every time.

SGLang was earliest on this path. At the end of 2023 they were already guessing what agent workloads would look like, turning context in VRAM into a tree so shared prefixes are automatically reused; in the paper it is called radix attention. Looking back this year, they guessed right. Their claim is that cache hit rate rose from 95% to 99%, and cost fell fivefold, because what is saved is four times the compute, not four percentage points. The HiCache tiered caching setup cut first-token latency by 56%.

Another front is scheduling. GPUs keep getting faster, CPUs have not, and preparing a batch the old way takes long enough to leave the GPU idle. They changed it so synchronization is deferred by one round, and batches run back to back. On Hopper it gives a 1.1x improvement, and on Blackwell's GB racks it can reach 2x. DeepSeek's newly open-sourced DeepInfer also uses this zero-overhead scheduling.

SSMs were sentenced to death back then; now they are in Kimi

Dan Fu is the head of research at Together, and he recounted an experience of being proven wrong. While doing his PhD he worked on SSMs, state space models, and the word in the field was that nobody would use this thing, it was a dead end, and Transformers would rule everything. Yet today the Kimi model has a layer called KDA, which is essentially an SSM linear attention layer. Sparse attention was also picked back up. Diffusion language models sounded even more absurd back then; now they are used as draft models for speculative decoding and are the best choice available.

What he found most absurd was the looped model: it loops tokens through the same layers several times, the weights stay unchanged, but the training compute goes up. At the time, his lab thought this idea was weird and that nobody would use it—then they turned around and found that both OpenAI and Anthropic were already using it. That line of his was quite inspiring: nowadays, once a good idea finishes its experiments, it really does show up in GPT-6 and GPT-7; the time from paper to launch is frighteningly short.

There are roughly three paths to saving money on inference. The model architecture path is the most dramatic: MoE activates only a subset of experts, sparse attention and MLA compress along the sequence, and on top of that there is quantization and distillation—this is where the orders of magnitude that can be cut are. The second is kernels and runtime: the batch of kernels like flash attention and paged attention, plus radix attention brought over from databases and operating systems. The third lies outside the model: managing routing, managing speculative decoding, managing context compression. Dan Fu said they simply let an agent search for how to write kernels on its own, running it as a self-improving loop. That project was originally called Together Compiler; he later realized it is not actually a compiler, and it is being renamed.

The same model, switch providers and it drops by seventy percent

Ollama CEO Jeff gave an even more unsettling talk. Everyone loves to compare how far open models lag behind closed ones, but when you actually measure it, for the same open model, the gap between the worst provider and the best provider is larger than the gap between open and closed. In 2024 someone tested 31 Llama endpoints; the weights were supposed to be identical, yet some outputs were closer to GPT-3.5 than to another Llama.

The causes listed are all trivial things. Overdone quantization, JSON parsing errors in tool calls, wrong weight versions, context windows secretly truncated, and even models registered under the wrong name—you think you are calling A, but it is actually B. In chat you cannot tell, but put it in an agent and it is fatal. A 10% deviation per step, run for 15 steps, and an ordinary coding task ends up with up to a 79% chance of being wrong.

So Moonshot simply released a Kimi K2 tool-calling leaderboard, so whoever has inaccurate parsing can go look for themselves, and Minimax followed with the same thing. Together's approach is more crude: have the model write Flappy Bird over and over in a loop, and once quantization goes wrong, the bird stops flying. They helped Minimax do NVFP4 quantization for M3; Minimax's own recipe could not match FP8 quality, but Together's did. This directly determines whether that model can make money.

Jeff also mentioned an easily overlooked direction. Today's Nvidia cards come with a trusted execution environment that can produce a proof for the weights and inference stack that are running, equivalent to letting the provider verify its identity on the spot. He said this is very cutting-edge, and once this path works, what is being sold is trust.

38 dollars per card-hour, and gross margin is still 80%

The day before the recording, DeepSeek open-sourced its own inference stack, DeepInfer. The accompanying claim is 38 dollars per card per hour, and when YC crunched the released numbers, the gross margin was 80%. That 80% is squeezed out of the system. Get the input cache hit rate to 96%, keep the KV cache for a full 72 hours, and then use expert parallelism to saturate a 72-card rack. The cost is also very real: if one card dies, there is no fallback, and the compute of all 72 cards is gone together.

The volume on the open-source side is already frightening. Together says they run 400 trillion tokens a month, one-eighth of Google's; over half the tokens on OpenRouter go through open models, and one day it even surpassed OpenAI. For models of the same quality, the price can be up to ten times cheaper. Kimi's 3-trillion-parameter model uses half a byte per parameter, so the weights alone are 1.5 TB, while a single B200 has only 280 GB of memory; it can only be sliced up and spread across a pile of cards, and how to slice it is itself a matter of money.

Back to that 12 times. Inference compute really is twelve times training, so the outcome of this race mainly depends on who can sell tokens more cheaply without errors; training a stronger model is merely the entry ticket. DeepSeek open-sourcing its stack and laying its gross margin on the table is tantamount to setting a price for the whole industry. The remaining question worth asking is: when inference becomes a standard business at 38 dollars per card-hour, how much longer can charging by intelligence hold up.

More hardcore podcast content: Podwise