Perplexity and NVIDIA propose JOLT self-distillation training method
Perplexity and NVIDIA collaboratively propose a post-training method, JOLT: it lets the same model serve as both teacher and student, and trains the teacher to learn from the rewards of its own attempts while penalizing drift away from the student's behavior. On the MATH benchmark, JOLT achieves accuracy comparable to GRPO with 6.5 times fewer training tokens; JOLT+ additionally provides the student model with outcome-based rewards, clearly surpassing GRPO on math, coding, terminal tasks, and tool-use benchmarks.
View original postNew research: We developed JOLT, a post-training method where one model acts as both teacher and student, with the teacher trained to give feedback the student can use. On MATH, it matched GRPO's accuracy with 6.5x fewer training tokens. https://t.co/KFHdZ0tuFn https://t.co/Y4Svd9QK4G
View original
New research: We developed JOLT, a post-training method where one model acts as both teacher and student, with the teacher trained to give feedback the student can use. On MATH, it matched GRPO's accuracy with 6.5x fewer training tokens. https://t.co/KFHdZ0tuFn https://t.co/Y4Svd9QK4G
Body
Sharing new post-training research on self-distillation + RL (JOLT), done together with @NVIDIAAI.
In self-distillation, the same model performs the role of both a teacher and a student. The teacher gets extra information that the student doesn't, and the student learns from feedback provided by the teacher. However, the teacher could still use shortcuts in ways the student has no way to follow.
Our proposed approach, JOLT, trains the teacher as well, to learn from rewards on its own attempts, while penalizing any drift that's far off from what the student model would do. JOLT+ further supplements JOLT by providing the student model outcome-based rewards in addition to the teacher supervision signals.
On the MATH benchmark, we found that JOLT matches GRPO with 6.5x fewer training tokens, while JOLT+ convincingly beats GRPO across math, coding, terminal tasks, and tool use benchmarks.