卡内基梅隆论文：预训练、微调与RL可统一为单一训练规则
The separate stages of AI training can collapse into a single method.
卡内基梅隆大学论文提出，预训练、微调与RL可共享同一训练规则，混合各阶段数据往往优于分阶段训练。作者将MaxRL重构为对模型自身正确答案的似然训练，使一个训练器能按任意比例混合预训练文本、精选数据与RL样本。在0.76B模型上，数学RL阶段混入5% midtraining数据后MBPP+达32.69%，分阶段训练为29.81%。
正文
The separate stages of AI training can collapse into a single method.
New Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can share a single training rule, and blending their data often beats separate stages.
LLMs are usually trained in stages: pretraining, midtraining, fine-tuning, then RL, each with its own algorithm. Hard switches between stages can also erase earlier skills.
The authors recast a form of RL called MaxRL as likelihood training on the model's own correct answers. That lets one trainer mix pretraining text, curated data, and RL samples in any proportion over time.
On reasoning puzzles with a 124M model, the best blended runs beat the best staged run and kept general skills better. After later coding RL, a 0.76B model that blended in 5% midtraining data during math RL reached 32.69% on MBPP+, versus 29.81% for staged training.
Treat your training pipeline as a data-mixing schedule, and try blending earlier data into RL instead of switching stages abruptly.