面壁智能 OpenBMB 推出 Diffusion Reward Models（DRM）
A reward of “3” can mean two completely different things.
面壁智能 OpenBMB 提出 Diffusion Reward Models（DRM），不再把人类偏好压缩成单一分数，而是学习完整奖励分布，保留分歧、不确定性与多种合理判断。DRM 可用分布不确定性识别不稳定的奖励决策，分布感知排序在 Best-of-N 选择上优于直接取均值；固定回复多次采样奖励分布还能带来新的 test-time scaling 维度。作为训练期奖励时，DRM 在 RLHF 中相比标量奖励基线提升下游策略表现。
正文
当前显示原文，中文全文译文尚未就绪。
A reward of “3” can mean two completely different things. Everyone thinks a response is mediocre — or half the people love it while the other half hate it. Most Reward Models cannot tell the difference. Introducing Diffusion Reward Models (DRM): instead of collapsing human preference into a single score, DRM learns the full reward distribution, preserving disagreement, uncertainty, and multiple plausible judgments. ✨ Paper:https://t.co/wS1fmHBeRV 🤗 Models: https://t.co/RGTU759A5y 💻 GitHub: https://t.co/KAow6DcoNn
Why it matters: Human disagreement is structured, not just noise. On datasets with repeated annotations, judgments often form separated or polarized patterns. More importantly, as human disagreement increases, DRM’s learned reward distribution becomes increasingly multimodal.
The distribution is useful, not just descriptive. DRM can use distributional uncertainty to identify unstable reward decisions, and distribution-aware ranking improves Best-of-N selection beyond simply taking the mean reward.
Reward Models get their own test-time scaling. Instead of only spending more compute on generating more responses, DRM can keep the response fixed and sample its reward distribution more times. More reward samples give a more reliable estimate — a new scaling axis unavailable to deterministic scalar RMs.
And the gains survive RLHF. When used as the training-time reward, DRM improves downstream policy performance over scalar reward baselines, showing that the benefit is not limited to offline RM benchmarks.
The takeaway: Reward modeling may lose something important when it compresses every human judgment into one number. “Everyone thinks this is average” and “people strongly disagree about this” should not look identical to a Reward Model. DRM makes that difference visible — and usable.