Meta paper finds that RL post-training reduces task coverage under multiple attempts

Understand this story

AI overview

Meta published a paper studying the impact of reinforcement learning post-training on agents' performance across multiple attempts. The study tested 14 pairs of base models and post-trained models on 3 agent benchmarks, and built the PTGS sampler, which raises the temperature for difficult prompts and lowers it for easy prompts. The results show that post-training makes agents perform better on their first attempt, but they solve fewer tasks that are solvable within up to 128 attempts. On WebShop, the gemma-4-31B base model solved over 85% of tasks within 128 attempts, while the post-trained version solved 56%. The Sharpening Tax proposed by the authors is positive in 36 of 42 model-benchmark combinations, meaning coverage shrinks. The authors suggest that when selecting models for retryable agents, one should measure how many tasks the model can ultimately solve, rather than looking only at first-attempt accuracy.

AI-generated from public reports

Latest development · ReportOct 30 01:14

Meta paper finds reinforcement learning post-training improves first-attempt success rate but reduces coverage across multiple attempts

Developments

1 confirmed developments

Oct 30· 1 reports
Meta paper finds RL post-training reduces task coverage across multiple attempts
Source:Rohan Paul

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Oct 30

01:14
Rohan Paul
Meta paper finds reinforcement learning post-training improves first-attempt success rate but reduces coverage across multiple attempts
A new Meta paper finds that reinforcement learning post-training makes agents perform better on first attempts, but fewer tasks can be solved within up to 128 attempts. The study tested 14 sets of base models and post-trained models on 3 agent benchmarks, and built the PTGS sampler, which raises temperature for difficult prompts and lowers it for easy prompts. On WebShop, the gemma-4-31B base model solved over 85% of tasks within 128 attempts, while the post-trained version solved 56%; the authors' proposed Sharpening Tax was positive in 36 of 42 model-benchmark combinations, meaning coverage shrank. The authors suggest that when selecting models for retryable agents, one should measure how many tasks the model can ultimately solve, rather than looking only at first-attempt accuracy.