Arena 如何为图像模型后训练设计复合奖励
How to design rewards for post-training frontier image models?
Arena 提出用复合奖励对前沿图像模型做后训练：在约 560 万对人工投票训练的 Bradley-Terry 奖励模型之外，加入由视觉语言模型评估的忠实度奖励、约束奖励和反奖励作弊评分。后训练后的 FLUX.2-dev 在实时 T2I 榜单提升 69 Elo 至 1202，Ideogram 4 提升 20 Elo 至 1224，超过所有公开开源模型。以 Gemini 3.5 Flash 为评判的离线消融显示胜率升至 64.2%，权重空间集成后进一步达 66.0%。
正文
当前显示原文，中文全文译文尚未就绪。
How to design rewards for post-training frontier image models?
Our research suggests human preference reward is necessary, but insufficient: A preference model may still reward outputs that look appealing but miss details, introduce unrequested content, or exhibit other forms of reward-hacking.
We therefore optimize towards a composite reward: - Bradley-Terry reward model trained on ~5.6M pairwise human votes - Faithfulness reward from auto-generated prompt checklists evaluated by a vision-language model - Constraint reward covering explicit and implicit user intent - Anti-reward-hacking rubric rewards targeting failures such as garbled text and photorealism drift This post-training recipe improves two already-strong open image models: - Post-trained FLUX.2-dev gains 69 Elo points on our live T2I leaderboard, scoring 1202 - Post-trained Ideogram 4 gains 20 Elo points reaching a score of 1224 and surpassing all publicly listed open models (as of Sep 04, 2026).
Offline ablations with Gemini 3.5 Flash as the judge, show that these reward components are complementary: win rate against the base model increases as we add faithfulness and then constraint rewards on top of preference-only training, reaching 64.2%.
Finally, we ensemble policies trained with and without the anti-reward-hacking objective directly in weight space, further increasing win rate to 66.0%.