Stanford CS336 Lecture 10: Percy Liang on Inference Optimization
English text is being prepared.
Stanford CS336 Lecture 10 "Inference" is taught by Percy Liang, who uses a single mathematical thread to connect three layers of inference optimization—architecture, algorithms, and systems: at the architecture layer, GQA/MLA/CLA reduce dimensionality, local attention truncates context, and quantization, pruning, and distillation compress the KV cache and parameters; at the algorithm layer, speculative sampling achieves lossless acceleration; at the systems layer, continuous batching, paging, prefix sharing, and copy-on-write are introduced. The lecture notes are in executable form, with formulas derived step by step by sympy and verified with assertions. Percy believes that the KV cache is the structural root of the Transformer's unfriendliness to inference, and that architectures natively designed for inference (SSMs, linear attention, diffusion-style non-autoregressive generation) may offer the greatest room for improvement.
View original postBeyond the five AI courses from three top universities—Stanford, CMU, and MIT—here are two more courses that lean more toward theory and fundamentals. CS336: Language Modeling from Scratch The capstone implementation course in Stanford's AI course chain, co-taught by Percy Liang and Tatsu Hashimoto, in which you build a complete language model from scratch by hand. CS221: Artificial Intelligence: Principles and Techniques Stanford's undergraduate core introductory AI course, taught by Percy Liang, which walks through the entire classical AI field using a unified mathematical framework (learning, search, MDPs, games, Bayesian networks, logic). Both courses come from Stanford University, and both instructors include @percyliang (professor at @stanfordnlp, co-founder of @togethercompute). CS336 course address https://t.co/weu3x3Ulst https://t.co/yq6XzfaqE4 CS221 course address https://t.co/aXReeC4gZQ https://t.co/f7d6VlTbXz
View original
Stanford、CMU 和 MIT 三所顶级大学的 5 门 AI 课程之外，再分享两门更偏理论和基础的课程。 CS336: Language Modeling from Scratch Stanford AI 课程链的顶点实现课，由 Percy Liang 与 Tatsu Hashimoto 联合执教，从零亲手构建一个完整的语言模型。 CS221: Artificial Intelligence: Principles and Techniques Stanford AI 的本科主干入门课，由 Percy Liang 执教，用统一的数学框架（学习、搜索、MDP、博弈、贝叶斯网络、逻辑）把整个经典 AI 领域走一遍。 这两门课程都来自 Stanford 大学，讲师都有 @percyliang (@stanfordnlp 教授，@togethercompute 联合创始人）。 CS336 课程地址 https://t.co/weu3x3Ulst https://t.co/yq6XzfaqE4 CS221 课程地址 https://t.co/aXReeC4gZQ https://t.co/f7d6VlTbXz
Body
斯坦福大学 CS336: Language Modeling from Scratch
第 10 讲「Inference」，在前后两讲 Scaling Laws 和 Scaling Laws II 之间，@percyliang 教授用一条完整的数学主线串起从架构、算法到系统的全部优化手段，把 “训练完的模型，如何又快又准地生成响应？” 讲得非常清楚！
讲义采用 CS336 特色的“可执行讲义”形式，所有公式都是符号计算（sympy）逐步推导并断言验证的。整讲的逻辑顺序是：自回归 → 无法沿序列并行 → 算术强度低 → memory-bound → 一切优化 = 减少每步搬运的字节数，围绕这个链路展开。
1. 架构层（有损）：GQA/MLA/CLA 降维、局部注意力截断、量化、剪枝蒸馏；全部服务于“把 KV cache 和参数压小，且用评测证明没伤准确率”；
2. 算法层（无损）：投机采样，用“验证快于生成”的不对称性换取精确的免费加速；
3. 系统层：连续批处理、分页、前缀共享、写时复制；操作系统经典思想向 GPU 服务的迁移；
4. 张力贯穿始终：延迟 vs 吞吐（批大小）、速度 vs 准确率（每一项有损技术）、新颖性 vs 可信度（GQA 与 MLA 论文结论的相互打脸）。
Percy 在结尾留下的展望是最本讲有分量的判断：KV cache 本身就是 Transformer 对推理不友好的结构性根源；一个从第一天起就为推理设计的架构（SSM、线性注意力、扩散式非自回归生成）可能是这一领域最大的改进空间。