唐杰复盘 GLM-5.3 Infra Agent 两周内优化推理基础设施
Two weeks.
唐杰（jietang）复盘了 GLM-5.3-Flash 从在国内加速器上首次跑通到承载全部生产流量仅用两周的过程，端到端吞吐提升 3.2 倍，其中大量工作由基于 GLM-5.3 的 Infra Agent 完成。他指出智能体卡住的原因往往不是写不出代码，而是不知道性能为何变差，因此团队构建了可被直接调用的分层验证接口，即密集反馈，包括正确性、系统行为和性能三类信号。智能体据此发现并修复了 KDA 上下文并行路径的 TF32 精度漂移（修复已合入 Flash Linear Attention PR #1180）、KV 传输未与 DeepEP dispatch 重叠导致的开销（从超过 30% 降至 1% 以下），以及 decode kernel 重复计算归一化的问题（提速 1.71 倍）。他强调人类仍负责定义目标、搭建反馈环境和审查高风险改动，但工程师的角色正从解决问题转向设计反馈。
查看原帖We’re sharing how GLM-5.3 helped build and optimize the inference infrastructure serving GLM-5.3-Flash. The system went from its first successful run to production readiness in less than two weeks, with end-to-end throughput tripling relative to the initial baseline. The key was dense feedback: local correctness tests, execution traces, microbenchmarks,…
展开完整原文
We’re sharing how GLM-5.3 helped build and optimize the inference infrastructure serving GLM-5.3-Flash. The system went from its first successful run to production readiness in less than two weeks, with end-to-end throughput tripling relative to the initial baseline. The key was dense feedback: local correctness tests, execution traces, microbenchmarks, and end-to-end measurements that enabled targeted hypothesis testing rather than reliance on aggregate performance metrics alone. https://t.co/yUf6OpJD7c
正文
当前显示原文，中文全文译文尚未就绪。
Two weeks.
That's how long it took to go from GLM-5.3-Flash's first run on domestic accelerators to serving all of its production traffic, with 3.2× end-to-end throughput along the way.
What I keep thinking about is who did much of the work: an Infra Agent powered by GLM-5.3.
A model helping optimize the system that serves it.
The conditions were hard. Limited memory and interconnect bandwidth. 1M-token context. Multimodal requests. An immature software stack where kernels were missing and documentation was often guesswork. Every optimization was a trade: compute for memory (ReplaySSM), communication for memory (intra-node tensor parallelism), precision for capacity (mixed INT8/FP8/BF16 caching), and disaggregation for scheduling freedom (Encode–Prefill–Decode).
But the most important lesson wasn't about any single optimization.
When the agent got stuck, it was rarely because it couldn't write the code. It was because it didn't know *why* things got worse.
"Throughput down 20%" tells you something broke. It doesn't tell you which layer, which hypothesis, or what to test next. In RL terms, it's a sparse reward with a credit assignment problem. And an end-to-end benchmark that takes hours makes exploration painfully slow.
Senior engineers solve this with an implicit process reward in their heads. They know when to check the timeline, when to run a microbenchmark, and which layer's output to compare.
So we made that explicit. We call it dense feedback: layered verification interfaces the agent can call directly.
Correctness feedback: did it compute right? System behavior feedback: where did the time go? Performance feedback: which option wins, under which conditions?
Each signal has to be local, cheap, and objectively verifiable.
Three things the agent found:
First, precision drift in KDA's context-parallel path that grew with sequence length. The cause was TF32 rounding error compounding through chained state-matrix merges. The fix is now merged upstream in Flash Linear Attention (PR #1180).
Second, KV transfer never overlapped with DeepEP dispatch. The agent followed the call chain across the Python/C++ boundary and found that the intranode path never released the GIL. After the fix, transfer overhead fell from over 30% to under 1%.
Third, a decode kernel recomputing the same normalization four times because of how it was chunked. The agent restructured it and got a 1.71× speedup. The idea came from "optimization skeletons" it had distilled by reading existing kernels across SGLang, FLA, and DeepGEMM.
To be clear about the boundaries: humans still defined the goals, built the feedback environment, and reviewed every high-risk change.
But the engineer's role is changing, from the person who solves the problem to the person who designs the feedback.
There's a deeper implication too. A layered, verifiable feedback environment built on real infrastructure tasks is exactly what training the next generation of models needs most. Every task the agent completes can become training ground for its successor.
We are still far from recursive self-improvement.
But the smallest loop now exists.
The model optimizes the system. The system serves the model.