CMU Paper: Pretraining, Fine-Tuning, and RL Can Be Unified into a Single Training Rule
A Carnegie Mellon University paper proposes that pretraining, fine-tuning, and RL can share the same training rule, and that mixing data from various stages often outperforms staged training. The authors reframe MaxRL as likelihood training on the model's own correct answers, enabling a single trainer to mix pretraining text, curated data, and RL samples in arbitrary proportions. On a 0.76B model, mixing 5% midtraining data into the math RL stage raised MBPP+ to 32.69%, compared with 29.81% for staged training.
Body
The separate stages of AI training can collapse into a single method.
New Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can share a single training rule, and blending their data often beats separate stages.
LLMs are usually trained in stages: pretraining, midtraining, fine-tuning, then RL, each with its own algorithm. Hard switches between stages can also erase earlier skills.
The authors recast a form of RL called MaxRL as likelihood training on the model's own correct answers. That lets one trainer mix pretraining text, curated data, and RL samples in any proportion over time.
On reasoning puzzles with a 124M model, the best blended runs beat the best staged run and kept general skills better. After later coding RL, a 0.76B model that blended in 5% midtraining data during math RL reached 32.69% on MBPP+, versus 29.81% for staged training.
Treat your training pipeline as a data-mixing schedule, and try blending earlier data into RL instead of switching stages abruptly.