Perplexity 与 NVIDIA 提出 JOLT 自蒸馏训练法
Sharing new post-training research on self-distillation + RL (JOLT), done together with @NVIDIAAI.
Perplexity 与 NVIDIA 合作提出后训练方法 JOLT：让同一模型同时充当教师和学生，并训练教师从自身尝试的奖励中学习，同时惩罚偏离学生行为的漂移。在 MATH 基准上，JOLT 以 6.5 倍更少的训练 token 达到与 GRPO 相当的准确率；JOLT+ 额外给学生模型提供基于结果的奖励，在数学、编程、终端任务和工具使用基准上明显超过 GRPO。
查看原帖新研究：我们开发了 JOLT，一种后训练方法，让同一个模型同时充当教师和学生，并训练教师给出学生能够利用的反馈。 在 MATH 上，它以 6.5 倍更少的训练 token 达到了与 GRPO 相当的准确率。 https://t.co/KFHdZ0tuFn https://t.co/Y4Svd9QK4G
查看原文
New research: We developed JOLT, a post-training method where one model acts as both teacher and student, with the teacher trained to give feedback the student can use. On MATH, it matched GRPO's accuracy with 6.5x fewer training tokens. https://t.co/KFHdZ0tuFn https://t.co/Y4Svd9QK4G
正文
当前显示原文，中文全文译文尚未就绪。
Sharing new post-training research on self-distillation + RL (JOLT), done together with @NVIDIAAI.
In self-distillation, the same model performs the role of both a teacher and a student. The teacher gets extra information that the student doesn't, and the student learns from feedback provided by the teacher. However, the teacher could still use shortcuts in ways the student has no way to follow.
Our proposed approach, JOLT, trains the teacher as well, to learn from rewards on its own attempts, while penalizing any drift that's far off from what the student model would do. JOLT+ further supplements JOLT by providing the student model outcome-based rewards in addition to the teacher supervision signals.
On the MATH benchmark, we found that JOLT matches GRPO with 6.5x fewer training tokens, while JOLT+ convincingly beats GRPO across math, coding, terminal tasks, and tool use benchmarks.