AI News

Trending story

Base Labs studies whether gradient-aligned rewards can achieve online policy supervision

2 reports2 story updatesUpdated Nov 14 08:24

Understand this story

AI overview

On November 14, 2026, Baseten Base Labs published a study exploring whether gradient-aligned rewards can turn supervised signals into on-policy ones. The study proposes using the cosine similarity between the log-likelihood gradients of student rollouts and expert demonstrations as a reward, which in theory combines the generality of on-policy reinforcement learning and SFT. The study also points out that this reward, like SFT, still tends to favor the specific wording of expert examples and is subject to the many challenges of distillation. In terms of implementation, the study uses tensor tricks to compress the full reward computation into about two standard backpropagations, with the cost mainly on the trainer side. There have been no further developments so far.

AI-generated from public reports

Latest development · Report

Can gradient-aligned rewards turn supervision into on-policy?

Developments

1 confirmed developments

  1. Can Base Labs research whether gradient-aligned rewards can achieve online policy supervision?

    Source:Baseten Base Labs: ModelsResearch

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 14

  1. Baseten Base Labs: Model ResearchFirst-party

    Can gradient-aligned rewards turn supervision into on-policy?

    Baseten Base Labs explores whether gradient-aligned rewards can turn the supervision signal into on-policy: using the cosine similarity of the log-likelihood gradients of student rollouts and expert demonstrations as the reward, which in theory combines the generality of on-policy RL and SFT. The authors point out that this reward will still, like SFT, be biased toward the specific wording of the expert examples, and is subject to the many challenges of distillation. In implementation, tensor tricks compress the full reward computation into about two standard backpropagations, with the cost mainly on the trainer side.

  2. Baseten Base Labs: Model ResearchFirst-party

    Can gradient-aligned rewards turn supervision into on-policy?

    Baseten Base Labs explores whether gradient-aligned rewards can make the supervision signal on-policy: using the cosine similarity of the log-likelihood gradients between student rollouts and expert demonstrations as the reward, which theoretically combines the generality of on-policy RL and SFT. The authors point out that this reward, like SFT, still favors the specific wording of expert demonstrations and is subject to the many challenges of distillation; with tensor tricks, the full reward can be computed efficiently in about two standard backpropagations.

Related stories

  1. Related storiesThinking Machines Lab introduces the Inkling 2 Small end-to-end 4-bit training recipe
  2. Related storiesAi2, in collaboration with multiple institutions, releases the XYBench benchmark test
  3. Related storiesZyphra proposes sparse attention method ZHA
  4. Related storiesApple proposes an online reinforcement learning UI navigation framework for real-time iOS applications