Chelsea Finn：生成式价值分类器是 Q 函数的新参数化
Generative value classifiers are a new parametrization of Q-functions that don't require action conditioning.
Chelsea Finn 介绍「生成式价值分类器」：一种无需动作条件化的 Q 函数新参数化方式。她认为这类方法或能与大型预训练策略良好结合，并从更丰富的输出信号中更好地学习和泛化。所引帖子指出，机器人策略无法像语言模型那样直接给输出打分，通常需从稀疏奖励单独训练 critic。
查看原帖在语言模型中，把生成模型变成评论家很简单：传入输出并提示给出分数。机器人策略做不到这一点——它们输出的是物理动作，而且没有内置的词汇表来预测分数。 因此，当涉及机器人策略上的强化学习时，我们通常从稀疏奖励中单独训练一个评论家——它对策略从预训练数据的动作中学到的一切一无所知。 我们认为这搞反了——评论家能否像策略那样从这些动作中学习？（1/n）
查看原文
In language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores. So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data. We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)
正文
Generative value classifiers are a new parametrization of Q-functions that don't require action conditioning.
We think these may combine gracefully with large, pre-trained policies and learn/generalize better from a richer output signal.
Paper: https://t.co/pNphW8m0uf