MIT "How to AI an AI" Lecture 6: From PPO and GRPO to On-Policy Distillation…
The lecture slides for Lecture 6, "RL & Self-Distillation in Practice," of MIT's Fall 2026 course MAS.S62 "How to AI an AI" are taught by Idan Shenfeld of Thinking Machines Lab, and formulate PPO, GRPO, on-policy distillation, and self-distillation as a single loss, differing only in the source of the advantage values. The slides break down the five stages of a single training step, and using Qwen3-8B as an example, give the order of magnitude of memory: about 16 GB for bf16 weights on the inference side, and about 16 bytes per parameter on the training side, totaling about 130 GB; they also compare two frameworks: verl for offline training and reef for learning while serving.
Why read this
AI score 55
The lecture slides place PPO, GRPO, and on-policy distillation within a single loss framework, and provide memory accounting and a monitoring checklist to help understand the engineering constraints of RL training.
Topics
Body · AI translation
RL & Self-Distillation in Practice
MIT's Fall 2026 course MAS.S62, "How to AI an AI for (Almost) Anything," also known as Self-Evolving AI, is about AI that can evolve itself. On November 4, week 9 moves into the second lecture of the "model evolution" unit, and these slides by Ao Qu are titled Lecture 6: RL & Self-Distillation in Practice.
The guest instructor for this class, @IdanShenfeld, comes from Thinking Machines Lab.
It covers concrete operations: how to treat a large language model as a policy, how advantage values are computed, why during training the model must sit on both the inference engine side and the training engine side, which curves to watch during training, and how to go from the offline training framework verl to reef, which learns while serving. The same week's reading also includes self-play, but these slides deal almost exclusively with the practice of reinforcement learning and distillation.
One loss, two ways of scoring
First, view the language model as a policy. The prompt is the starting point, each generated token counts as one step of action, the state is "the prompt plus what has already been generated," and the reward is usually given only once after the entire response ends. The general form of the policy gradient is:
A_t is called the advantage, indicating whether this token or this response is better or worse than expected. PPO and GRPO also multiply by the probability ratio between the new and old policies, then use clipping to limit the size of each step's update, avoiding going too far in one step.
The difference between the two lies in how the advantage is computed:
PPO additionally trains a value model V, which estimates "roughly how many more points can still be obtained from here," and the advantage is the actual return minus this estimate.
GRPO (from DeepSeekMath) samples multiple responses to the same question and standardizes the rewards using the mean and standard deviation within the group, requiring no value model.
On-policy distillation (OPD) and self-distillation can also be written as the same loss, except that A_t is replaced by "the teacher's log probability for this token minus the student's own log probability":
In OPD the teacher is another, larger model, e.g., a large model teaching a small model.
In self-distillation the teacher is the same model, except that the prompt includes an extra condition c, such as a demonstration, a standard solution, or human feedback. Without this condition, the teacher and student are identical, and there is no learning signal.
The slides compare the information content of the two in one sentence: reinforcement learning usually shares one reward across the entire response, giving only a constant amount of information per episode; OPD gives every token a score, so the information grows with length N. Reinforcement learning moves toward high rewards and may learn to game the score; OPD moves toward the teacher, and its ceiling is the teacher itself. The figures and arguments in this part are cited from Thinking Machines' On-Policy Distillation article, and the slides do not list separate evaluation scores.
One piece of human feedback, fed to both reinforcement learning and self-distillation
The slides give the example of OpenClaw-RL: a teaching-assistant agent grades homework, and a response is written too briefly, so someone corrects it with "you need to explain to students clearly why each step is correct."
On the reinforcement learning path: the judge model gives a score, e.g., r = -1, and the entire response shares the same advantage value.
On the self-distillation path: this correction is written into the prompt, and the same model is made to act as the teacher with it, yielding a separate advantage value for each token.
The two parts of the loss are added together, and one update is made using the same batch of responses. The former has a score every round but is very coarse; the latter can point out which words should be changed, but this fine-grained signal usually exists only when a correction appears. This is an existing method cited in the slides, not a new algorithm proposed in this lecture.
How one training step proceeds, and why the model must be kept in two copies
The slides break one training step into five stages:
Generate responses: use inference engines such as vLLM and SGLang, sampling n responses per prompt.
Score: use rules, unit tests, or a judge model, giving each response a score.
Compute advantage: reward minus baseline; in the case of distillation, replace it with the difference in log probabilities between teacher and student.
Update parameters: backpropagate on FSDP or Megatron.
Synchronize weights: copy the updated weights back to the inference engine.
The reason for splitting into two sides is that generation and training need different things: generation must be fast and relies on the KV cache; training must store gradients and optimizer states. So after every update, the weights must be synchronized once. Taking Qwen3-8B with about 8.2 billion parameters as an example, the magnitudes given in the slides are:
On the inference engine side: bf16 weights are about 16 GB, plus about 0.15 MB of KV cache per token.
On the training engine side: weights, gradients, and Adam optimizer states total about 16 bytes per parameter, i.e., around 130 GB, plus activations, which cannot fit on a single 80 GB GPU and must be split across multiple GPUs in parallel.
The batch is also divided into three levels. The example configuration in the slides is: 1024 prompts, 5 samples per prompt, for a total of 5120 responses; parameters are updated once every 256 prompts, so it is split into 4 updates; 8 GPUs perform data parallelism, and each GPU is further split into micro-batches to accumulate gradients.
There is a detail here that is easy to overlook: only the first update strictly counts as on-policy. Once the parameters are updated, the current policy is no longer the one used when sampling.
Two more terms can be added to the loss. One is a KL penalty relative to the initial model, which is often turned off in reasoning-oriented reinforcement learning and often turned on in RLHF. The other is an entropy bonus, used to encourage exploration, which is often turned off in practice, but the entropy curve must be watched.
Which curves to watch during training
The slides use verl's public mathematical reasoning baseline as an example: Qwen2.5-Math-7B plus a variant of GRPO, evaluated on AIME 2024, and on that basis organize six categories of monitoring metrics:
This is a monitoring checklist; it does not guarantee that you can reproduce the same final score, and the slides also do not write the accuracy at the endpoint of the curves as specific numbers.
With verl, the part you actually have to write
The slides compress the practical workflow into a few steps:
Organize the task into a parquet file, one prompt per line, optionally with a standard answer.
Write a compute_score function that extracts the answer from the response and returns 1 if it is correct and 0 if it is wrong.
Launch verl.trainer.main_ppo, and choose GRPO, the model, vLLM, the number of samples, the batch size, and the number of GPUs.
Watch the curves mentioned above.
After training ends, merge the FSDP shards, convert to Hugging Face format, and then deploy with vLLM.
In synchronous mode, the internal order of one training step is roughly: generate, group into batches, if necessary put the inference replica to sleep to free up GPU memory, balance the number of tokens across GPUs, compute the log probabilities of the old policy and the reference model, compute advantages, update the model, and finally synchronize the weights back to the inference engine. The slides provide simplified pseudocode corresponding to the HybridFlow and verl code paths.
From verl to reef: train well first and then go live, or learn while serving
verl: repeatedly generate, score, and update on a fixed prompt set you have prepared, and then deploy the exported model separately.
reef: the inference service itself is the learner, directly using online requests, responses, and feedback to train and evaluate, and when switching to a new version there is no need to restart the entire system.
The slides' summary is very direct: verl is a tool for training on a dataset, and reef is an inference server that learns from its own online traffic. This exactly corresponds to the infrastructure layer of the course's major theme of "self-evolving AI."
A few thoughts after reading
The most valuable part of these slides is that they bring "large models plus reinforcement learning" down from paper formulas to a hands-on engineering understanding: the same loss, only the source of the advantage values differs; the inference and training engines each have their own GPU memory accounting; the on-policy condition is actually very fragile; a monitoring checklist is much more useful than memorizing another new acronym.
It presents on-policy distillation and self-distillation as methods in the same form as reinforcement learning but with finer supervision, which is the key to understanding the recent route of "rely less on sparse rewards and use teacher probabilities more." As for the course's zero-data self-play, one still needs to read papers such as Absolute Zero and R-Zero separately.