Natural language is often the easiest and most convenient modality for humans
to specify tasks for robots. However, learning to ground language to behavior
typically requires impractical amounts of diverse, language-annotated
demonstrations collected on each target robot. In this work, we aim to separate
the problem of what to accomplish from how to accomplish it, as the former can
benefit from substantial amounts of external observation-only data, and only
the latter depends on a specific robot embodiment. To this end, we propose
Video-Language Critic, a reward model that can be trained on readily available
cross-embodiment data using contrastive learning and a temporal ranking
objective, and use it to score behavior traces from a separate reinforcement
learning actor. When trained on Open X-Embodiment data, our reward model
enables 2x more sample-efficient policy training on Meta-World tasks than a
sparse reward only, despite a significant domain gap. Using in-domain data but
in a challenging task generalization setting on Meta-World, we further
demonstrate more sample-efficient training than is possible with prior
language-conditioned reward models that are either trained with binary
classification, use static images, or do not leverage the temporal information
present in video data.

基于视频 - 语言评论家的奖励模型，可以在现有的跨体现数据上进行训练，使用对比学习和时间排序目标，对来自单独强化学习执行者的行为轨迹进行评分。在 Meta-World 任务中，通过在 Open X-Embodiment 数据上训练奖励模型，实现了比仅稀疏奖励模型更高效的策略训练，尽管存在显著的领域差异。使用 Meta-World 上的具有挑战性任务推广设置的领域数据进一步证明，相比先前使用二元分类训练的语言条件奖励模型，静态图像或不利用视频数据中的时间信息，我们的方法实现了更高效的训练。