Understand this story
On November 14, 2026, Baseten Base Labs published a study exploring whether gradient-aligned rewards can turn supervised signals into on-policy ones. The study proposes using the cosine similarity between the log-likelihood gradients of student rollouts and expert demonstrations as a reward, which in theory combines the generality of on-policy reinforcement learning and SFT. The study also points out that this reward, like SFT, still tends to favor the specific wording of expert examples and is subject to the many challenges of distillation. In terms of implementation, the study uses tensor tricks to compress the full reward computation into about two standard backpropagations, with the cost mainly on the trainer side. There have been no further developments so far.
AI-generated from public reports
Latest development · Report
Can gradient-aligned rewards turn supervision into on-policy?Developments
1 confirmed developments
Can Base Labs research whether gradient-aligned rewards can achieve online policy supervision?
Source:Baseten Base Labs: ModelsResearch
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 14
Baseten Base Labs: Model ResearchFirst-party
Can gradient-aligned rewards turn supervision into on-policy?
Baseten Base Labs explores whether gradient-aligned rewards can turn the supervision signal into on-policy: using the cosine similarity of the log-likelihood gradients of student rollouts and expert demonstrations as the reward, which in theory combines the generality of on-policy RL and SFT. The authors point out that this reward will still, like SFT, be biased toward the specific wording of the expert examples, and is subject to the many challenges of distillation. In implementation, tensor tricks compress the full reward computation into about two standard backpropagations, with the cost mainly on the trainer side.
Baseten Base Labs: Model ResearchFirst-party
Can gradient-aligned rewards turn supervision into on-policy?
Baseten Base Labs explores whether gradient-aligned rewards can make the supervision signal on-policy: using the cosine similarity of the log-likelihood gradients between student rollouts and expert demonstrations as the reward, which theoretically combines the generality of on-policy RL and SFT. The authors point out that this reward, like SFT, still favors the specific wording of expert demonstrations and is subject to the many challenges of distillation; with tensor tricks, the full reward can be computed efficiently in about two standard backpropagations.