Stanford CS336 Lecture 10: Percy Liang on Inference Optimization
Stanford CS336 Lecture 10 "Inference" is taught by Percy Liang, who uses a single mathematical thread to connect three layers of inference optimization—architecture, algorithms, and systems: at the architecture layer, GQA/MLA/CLA reduce dimensionality, local attention truncates context, and quantization, pruning, and distillation compress the KV cache and parameters; at the algorithm layer, speculative sampling achieves lossless acceleration; at the systems layer, continuous batching, paging, prefix sharing, and copy-on-write are introduced. The lecture notes are in executable form, with formulas derived step by step by sympy and verified with assertions. Percy believes that the KV cache is the structural root of the Transformer's unfriendliness to inference, and that architectures natively designed for inference (SSMs, linear attention, diffusion-style non-autoregressive generation) may offer the greatest room for improvement.
View original postBeyond the five AI courses from three top universities—Stanford, CMU, and MIT—here are two more courses that lean more toward theory and fundamentals. CS336: Language Modeling from Scratch The capstone implementation course in Stanford's AI course chain, co-taught by Percy Liang and Tatsu Hashimoto, in which you build a complete language model from scratch by hand. CS221: Artificial Intelligence: Principles and Techniques Stanford's undergraduate core introductory AI course, taught by Percy Liang, which walks through the entire classical AI field using a unified mathematical framework (learning, search, MDPs, games, Bayesian networks, logic). Both courses come from Stanford University, and both instructors include @percyliang (professor at @stanfordnlp, co-founder of @togethercompute). CS336 course address https://t.co/weu3x3Ulst https://t.co/yq6XzfaqE4 CS221 course address https://t.co/aXReeC4gZQ https://t.co/f7d6VlTbXz
View original
Stanford、CMU 和 MIT 三所顶级大学的 5 门 AI 课程之外，再分享两门更偏理论和基础的课程。 CS336: Language Modeling from Scratch Stanford AI 课程链的顶点实现课，由 Percy Liang 与 Tatsu Hashimoto 联合执教，从零亲手构建一个完整的语言模型。 CS221: Artificial Intelligence: Principles and Techniques Stanford AI 的本科主干入门课，由 Percy Liang 执教，用统一的数学框架（学习、搜索、MDP、博弈、贝叶斯网络、逻辑）把整个经典 AI 领域走一遍。 这两门课程都来自 Stanford 大学，讲师都有 @percyliang (@stanfordnlp 教授，@togethercompute 联合创始人）。 CS336 课程地址 https://t.co/weu3x3Ulst https://t.co/yq6XzfaqE4 CS221 课程地址 https://t.co/aXReeC4gZQ https://t.co/f7d6VlTbXz
Body · AI translation
Stanford CS336: Language Modeling from Scratch
Lecture 10, "Inference," sits between the two lectures Scaling Laws and Scaling Laws II. Professor @percyliang uses a single complete mathematical thread to connect all optimization methods from architecture and algorithms to systems, explaining very clearly: "How can a trained model generate responses quickly and accurately?"
The lecture notes use CS336's characteristic "executable lecture notes" format: all formulas are derived step by step via symbolic computation (sympy) and verified with assertions. The logical order of the whole lecture is: autoregression → cannot parallelize along the sequence → low arithmetic intensity → memory-bound → all optimization = reducing the number of bytes moved per step, and it unfolds around this chain.
1. Architecture layer (lossy): GQA/MLA/CLA dimensionality reduction, local attention truncation, quantization, pruning and distillation; all of these serve "shrinking the KV cache and parameters, and using evaluations to prove accuracy is not harmed";
2. Algorithm layer (lossless): speculative sampling, using the asymmetry of "verification is faster than generation" to obtain exact free speedup;
3. Systems layer: continuous batching, paging, prefix sharing, copy-on-write; the migration of classic operating system ideas to GPU serving;
4. Tensions run throughout: latency vs throughput (batch size), speed vs accuracy (every lossy technique), novelty vs credibility (the conclusions of the GQA and MLA papers contradicting each other).
The outlook Percy leaves at the end is the most weighty judgment of this lecture: the KV cache itself is the structural root of the Transformer's unfriendliness to inference; an architecture designed for inference from day one (SSMs, linear attention, diffusion-style non-autoregressive generation) may be the largest room for improvement in this field.