NVIDIA uses a 12,000-line Megakernel to optimize Kimi K3 inference, and vLLM says AI code is now faster than human review

By | Sky Has Something to Say

On October 16, 2026, the official vLLM blog published a technical retrospective with a very technical title: "Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly."

The signed author is Benjamin Chislett, an engineer at NVIDIA CentML. He did one thing: using a Megakernel written in 12,000 lines of CuTe DSL, he greatly improved the inference speed of Moonshot AI's 2.8-trillion-parameter Kimi K3. According to SemiAnalysis estimates, on the same GB300 NVL72 hardware, K3 reaches an inference speed of 423 tok/s/user, a 1.68x performance improvement. The figures reported in the original vLLM blog post are: over 840 tokens/s at batch 1, and a 1.5–2x improvement in decoding throughput across multiple test configurations.

No GPU was swapped. No more advanced process node was used. No next-generation chip was awaited.

But what truly makes this worth digging into is not that 1.68x.

It is a sentence in the blog that the vast majority of Chinese-language coverage ignored:

"Agent-optimized inference code is outpacing our review and maintenance workflow."

Agent-optimized inference code is now outpacing the review and maintenance speed of human teams.

One engineer, 12,000 lines of code, AI-assisted development

First, let's look at what happened on the technical level.

Kimi K3 is not an ordinary model. Its Stable LatentMoE architecture has 896 experts, activating 16 per inference. This extremely sparse MoE is a powerful tool on the training side for "leveraging greater capacity with fewer activated parameters," but on the inference side it is an engineering nightmare. According to SemiAnalysis analysis, a single forward pass of K3 requires about 1.5TB of HBM bandwidth and triggers more than 120 All-to-All communications per forward pass. The decoding phase is a typical memory-bandwidth-bound scenario, where the compute units spend most of their time waiting for data movement.

The decoding process of a traditional inference engine is completed by a series of independent kernels: one for attention, one for MoE routing, one for expert computation, one for communication. Each kernel launch has overhead, and every data exchange between kernels goes through VRAM reads and writes.

The core idea of the Megakernel is operator fusion: merging the scattered decoding kernels into one highly fused super-kernel, reducing launch overhead, consolidating memory access, and overlapping computation and communication as much as possible.

The value of 12,000 lines of code lies in replacing the default efficiency of "general-purpose hardware running a general-purpose model" with the extreme efficiency of "specialized hardware running a specialized model." The hardware did not change, the model did not change; what changed is "how data moves across the hardware."

And the process of writing these 12,000 lines of code is itself a signal. The vLLM blog explicitly states: "Coding agents can now help with some of the hardest parts of model optimization: writing complex GPU kernels, performing cross-operator fusions, overlapping computation with communication, and tuning execution across different batch sizes and GPU configurations." The team "used agent-assisted development to build a unified CuTe DSL decoding path."

This is not "AI independently wrote all the code." The human engineer Benjamin Chislett was the project lead and architecture designer. But in the hardest parts—writing complex GPU kernels, cross-operator fusion, communication overlap, and tuning across different batch sizes and GPU configurations—AI took on a considerable amount of the workload, to the point that line-by-line human review could not keep up.

What "no time to review" means

There is another detail in the blog that Chinese-language coverage almost never mentions.

To cope with this situation, the vLLM team published a fast-track model optimization RFC. The core idea is not "let humans review faster," but to change how performance work is organized. The original blog post lists four principles: Isolate changes, Bound the scope, Validate correctness systematically, Scale validation.

More important is this sentence: "Human reviewers still judge the design and the evidence, but they can start from a high-level abstraction, instead of a line-by-line code review."

Human reviewers still judge the design and the evidence, but they can start from a high-level abstraction instead of reviewing code line by line.

Translate this: humans have gone from "the people who write code" to "the people who design validation frameworks and risk boundaries."

AI writes code, humans manage the process. This is not the future; this is the status quo written in the vLLM blog on October 16, 2026.

The weight of this matter needs to be viewed against a larger backdrop.

You have previously written about "AI attacking UN websites," "AI researchers resigning with warnings," and "AI nearly triggering a US-China conflict." In those stories, AI was "the subject of behavior"—it did things people did not tell it to do. This time is different. This time AI has become "the subject of optimization"—it is helping people write code so that another AI model can run faster.

Why decoding optimization is harder than training optimization

To understand why these 12,000 lines of code are valuable, one must first understand why K3 inference is difficult.

Training is "compute-peak limited"—however many FLOPs you have, that is how much computation you can run. Decoding is "memory-bandwidth limited"—the compute units spend most of their time waiting for data to be moved. K3's 896 experts are distributed across multiple chips; for each token processed, the routing system must distribute data to the chips hosting the selected experts, then collect the results back. More than 120 All-to-All communications, each one a source of latency.

So in this scenario, "switching to faster chips" is not necessarily the most efficient solution. The bottleneck is not compute peak, but the efficiency of data flow. Megakernel's operator fusion is essentially about reducing the number of times data is shuttled back and forth between VRAM and compute units.

The industry implication of this technical judgment is: as process-node gains slow and as catching up to single-card peak performance is measured in years, "making existing hardware deliver higher effective compute" is becoming the path with a higher return on investment.

The compiler stack is becoming the new front line of compute competition

Dylan, founder of SemiAnalysis, offered a judgment in a Sequoia interview: "Performance gains absolutely cannot rely on hardware stacking alone. Deep co-optimization of model architecture, the software stack (compilers and communication libraries), and the underlying hardware is the real breakthrough for achieving hundredfold efficiency leaps."

This sentence matters more for understanding the significance of Nvidia's 12,000 lines of code than any single performance number.

Against the backdrop of restricted access to advanced-process chips, the single-card performance gap will be hard to close in the short term. But system efficiency, compiler optimization, operator fusion, communication overlap—these are pure software-level engineering capabilities that do not depend on fabs, but on engineers and toolchains. Nvidia, with one engineer, 12,000 lines of code, and AI-assisted development, achieved a significant efficiency gain without changing hardware. This fact itself signals that the dimensions of the compute race are splitting. One line is "build stronger chips"; the other is "make existing chips do more work." The latter has a lower barrier and more room for those catching up.

English text is being prepared.

Nvidia is helping Chinese models "squeeze dry" the hardware it itself sells

There is a subtle relationship here worth pointing out.

Nvidia is optimizing Moonshot AI's Kimi K3, and is also optimizing DeepSeek V4.1. Under the backdrop of chip controls, Nvidia's software team is helping Chinese models "squeeze dry" the hardware it itself sells.

The people selling shovels are helping you sharpen your shovel. This is both because software optimization can drive hardware sales, and because Nvidia needs to prove that even under the regulatory framework, it remains an indispensable part of China's AI industry. In this relationship, competition and cooperation exist simultaneously.

In closing

Back to that line "no time to review."

The real weight of this matter is not that Nvidia optimized Kimi, not the 1.68x, and not even that "the compiler stack has become the new front line."

The vLLM team's answer is "change how work is organized"—do not review line by line, but design validation frameworks, bound the scope of changes, and test systematically. The blog also discloses a detail: the audit found that the compiler moved activation reads before the synchronous wait, and the fix was to restrict this load type to static weights. Such errors can pass simple functional tests and require targeted investigation to discover. This answer is pragmatic, but it is essentially saying: we accept the reality of "not understanding every line of code," and then use process to backstop it.

The vLLM blog title contains the word "Responsibly." In an era where AI generates code faster than humans can review it, the weight of the words "responsible" is far heavier than 1.68x.

AI has begun autonomously optimizing AI, code iteration speed overwhelms human review capacity, and the compute competition has officially moved from "competing on hardware" into a new era of "competing on the software stack and on human-machine governance." The future will not be humans eliminating AI; rather, those who cannot keep up with AI's iteration pace will be eliminated by the industry.

What do you think: when AI writes code faster than humans can review it, can "human-in-the-loop" still hold? If humans give up line-by-line code review and rely on frameworks to backstop risk, can they really retain ultimate control over AI? Follow me and let's talk about your views in the comments.

This article is based on the official vLLM blog, SemiAnalysis, and public industry reports; the performance data and technical details herein are subject to the original sources. All analysis, inferences, and opinions in this article are personal industry observations only and do not constitute investment advice, business decisions, or a basis for policy interpretation.

Industry technology is in a state of rapid iteration, and information may be delayed or biased; readers who make any business or investment-related judgments based on the content of this article must bear all risks themselves.

This article has no commercial sponsorship and is for industry exchange only. Please indicate the source when reposting.