Jia-Bin Huang releases a 9-episode, 230-minute "AI Training and Inference" training-and-inference video series
Jia-Bin Huang (@jbhuang0604) produced the 9-episode, roughly 230-minute "AI Training and Inference" video series, covering the fundamentals of foundation model training and inference. Shao Meng went through the content of each episode in order: ZeRO and FSDP, five-dimensional parallelism (including the example of Llama 3's 16384-GPU layout), LoRA and its variants, speculative decoding, on-policy distillation, from PPO to GRPO, AdamW, and Muon and MuonClip; Episode 9, LLM Serving, is members-only content. Among these, fine-tuning a 70B-scale model on a single 48GB GPU with QLoRA, and Muon's compute efficiency being about twice that of AdamW, are both claims made by the speaker in the videos.
Why read this
AI score 63
This content strings the 9-episode training-and-inference course into a learning path in the order of memory, parallelism, fine-tuning, decoding, distillation, RL, and optimizers, making it easy to follow step by step.
Topics
View original postI'm building a playlist/course for LLM training and inference (WIP). https://t.co/0JdUqjiiQm So far, we have common topics like distributed training, fine-tuning, RL, OPD, serving, spec. decoding, and basic optimizers. What else should I include to make this more complete? 🤔
View original
I'm building a playlist/course for LLM training and inference (WIP). https://t.co/0JdUqjiiQm So far, we have common topics like distributed training, fine-tuning, RL, OPD, serving, spec. decoding, and basic optimizers. What else should I include to make this more complete? 🤔
Body · AI translation
9 episodes, 230 minutes. The @jbhuang0604 series "AI Training and Inference" explains the fundamentals of foundation model training and inference clearly, covering the entire training-and-inference architecture: single-GPU memory → thousand-GPU parallelism → cheap fine-tuning → inference acceleration → small-model distillation → inference RL → optimizers (AdamW→Muon)
The connecting logic of this training-and-inference series First solve "the model won't fit on one GPU," then use five-dimensional parallelism to scale to the ten-thousand-GPU level; then the LoRA family brings adaptation costs down; on the inference side, speculative decoding accelerates without changing the output distribution; capability compression goes through on-policy distillation; reasoning ability uses the RL line from PPO to GRPO; and finally it returns to the optimizer baseline and Muon. The serving side (KV cache, scheduling, etc.) is placed last, in episode 9. https://t.co/guGpu2DlaP
1. ZeRO & FSDP (about 23 minutes) How floating point and mixed precision consume GPU memory; data parallelism replicates optimizer states and gradients. ZeRO-1/2 shards the optimizer and gradients, ZeRO-3 further shards the weights; FSDP is the same sharding schedule implemented module-by-module in PyTorch.
2. Five-dimensional parallelism (about 40 minutes) When pure data parallelism is not enough, split along data / tensor / sequence·context / pipeline / expert. Tensor parallelism's row-column splitting and all-reduce; context parallelism and ring attention; pipeline from GPipe, 1F1B to zero-bubble / DualPipe; MoE's all-to-all; and using Llama 3's 16384-GPU layout as an example, giving a three-step "cheat sheet" for arranging parallelism.
3. LoRA and variants (about 21 minutes) Freeze W0W_0W0 and learn the low-rank ΔW=BA\Delta W=BAΔW=BA; initialization commonly uses random A and zero B. LoRA+ gives A/B different learning rates; QLoRA uses NF4 etc. to quantize the backbone while keeping adapters in high precision (the subtitles say it can fine-tune 70B-scale models on a single 48GB GPU, which is the speaker's statement); VeRA shares a frozen random basis and trains only scaling; DoRA separates magnitude and direction.
4. Speculative decoding (about 25 minutes) A small draft model guesses ahead, and the large model verifies in parallel; rejection sampling guarantees the same distribution as the target model. Then it expands on draft tree, Medusa, MTP, the EAGLE series, and DFlash / DFlash 2.
5. On-policy distillation (about 31 minutes) Soft labels and "dark knowledge" → sequence-level distillation; exposure bias; on-policy RL rewards are sparse. On-policy distillation: the student uses its own trajectories while still getting the teacher's dense guidance; and it emphasizes the KL direction; applications mention reasoning, personalization, data reuse, and self-distillation.
6. GRPO (about 23 minutes) From REINFORCE, baseline/value, GAE, Actor-Critic, TRPO/PPO (especially clip) to GRPO: sample a group of answers to the same question, use the within-group normalized reward as the advantage, share the advantage across the whole segment, and optionally skip training a separate value; add KL relative to the reference policy. Dr. GRPO removes length normalization and standard-deviation division, alleviating biases such as "getting it right makes it shorter, and hard ones get their advantage amplified." The video links this to DeepSeek-R1-style reasoning training.
7. AdamW (about 7 minutes) SGD → momentum → RMSProp → Adam; Adam entangles decay and adaptive step size, which hurts generalization; AdamW decouples weight decay and becomes the long-term default.
8. Muon / MuonClip (about 18 minutes) The momentum matrices of linear layers are nearly low-rank, so rare directions get weak updates; Muon uses Newton–Schulz etc. to orthogonalize the momentum matrix (the speaker says it is about twice as compute-efficient as AdamW, which is a claim made in the video). At large scale, pair it with decay and learning rates adjusted by matrix size; attention logit explosion is handled with QK-clip, extended to MuonClip in MLA scenarios.
9. LLM Serving (about 42 minutes) Members-only content; if after watching the first 8 episodes you feel you need it, friends can study it on their own.