Understand this story
On October 30, 2026, Baseten Base Labs released a study analyzing the computational efficiency of value functions in reinforcement learning, comparing the PPO critic with GRPO. Its toy experiments showed that at the same number of steps, the PPO critic performed on par with or worse than GRPO; and a "perfect critic" could at most eliminate only 40-60% of the advantage variance. Based on this, Base Labs concluded that a value function is worth adopting only when the gradient noise it eliminates exceeds its training cost, and the only scenario in which it might help is a small value head, but even then the benefit is merely marginal.
AI-generated from public reports
Latest development · Report
Analyzing the Computational Efficiency of Value Functions in RL: PPO critic vs. GRPO | Base LabsDevelopments
1 confirmed developments
Base Labs analyzes the computational efficiency of value functions in RL
Source:Baseten Base Labs: ModelsResearch
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 30
Baseten Base Labs: Model ResearchFirst-party
Analyzing the Computational Efficiency of Value Functions in RL: PPO critic vs. GRPO | Base Labs
Baseten Base Labs' toy experiments show that at the same number of steps, the PPO critic performs on par with or behind GRPO, while a "perfect critic" can eliminate at most 40-60% of the advantage variance. The authors argue that a value function is only worthwhile when the gradient noise it eliminates exceeds its training cost; the only scenario where it might help is with a small value head, but even then the gains are only marginal.