Recent work in visual representation learning for robotics demonstrates the
viability of learning from large video datasets of humans performing everyday
tasks. Leveraging methods such as masked autoencoding and contrastive learning,
these representations exhibit strong transfer to policy learning for visuomotor
control. But, robot learning encompasses a diverse set of problems beyond
control including grasp affordance prediction, language-conditioned imitation
learning, and intent scoring for human-robot collaboration, amongst others.
First, we demonstrate that existing representations yield inconsistent results
across these tasks: masked autoencoding approaches pick up on low-level spatial
features at the cost of high-level semantics, while contrastive learning
approaches capture the opposite. We then introduce Voltron, a framework for
language-driven representation learning from human videos and associated
captions. Voltron trades off language-conditioned visual reconstruction to
learn low-level visual patterns, and visually-grounded language generation to
encode high-level semantics. We also construct a new evaluation suite spanning
five distinct robot learning problems $\unicode{x2013}$ a unified platform for
holistically evaluating visual representations for robotics. Through
comprehensive, controlled experiments across all five problems, we find that
Voltron's language-driven representations outperform the prior
state-of-the-art, especially on targeted problems requiring higher-level
features.

本文介绍了一种基于语言驱动的视觉表示学习方法，即 Voltron，并通过全面的实验验证表明其在五种不同的机器人学习问题上的表现优于以往技术，特别是在需要更高级特征的问题上。