Lilian Weng：on-policy 蒸馏可作过程奖励模型
On-policy distillation provides an elegant way to use the teacher model as a process reward model to provide dense reward while preventing SFT style "OOD shock"
On-policy 蒸馏提供了一种优雅的方式，将教师模型用作过程奖励模型，在 rollout 过程中提供密集奖励，同时避免 SFT 式的“OOD shock”。
查看原帖Our latest post explores on-policy distillation, a training approach that unites the error-correcting relevance of RL with the reward density of SFT. When training it for math reasoning and as an internal chat assistant, we find that on-policy distillation can outperform other approaches for a fraction of the cost. https://t.co/JhpyWQOpBe
正文
当前显示原文，中文全文译文尚未就绪。
On-policy distillation provides an elegant way to use the teacher model as a process reward model to provide dense reward while preventing SFT style "OOD shock" during rollout.