Chelsea Finn: Generative value classifiers are a new parameterization of Q-functions
Chelsea Finn introduces "generative value classifiers": a new way to parameterize Q-functions without action conditioning. She believes such methods may combine well with large pretrained policies and learn and generalize better from richer output signals. The quoted post notes that robot policies cannot directly score outputs the way language models can, and typically require training a separate critic from sparse rewards.
View original postIn language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores. So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data. We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)
View original
In language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores. So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data. We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)
Body
Generative value classifiers are a new parametrization of Q-functions that don't require action conditioning.
We think these may combine gracefully with large, pre-trained policies and learn/generalize better from a richer output signal.
Paper: https://t.co/pNphW8m0uf