To build general robotic agents that can operate in many environments, it is
often imperative for the robot to collect experience in the real world.
However, this is often not feasible due to safety, time, and hardware
restrictions. We thus propose leveraging the next best thing as real-world
experience: internet videos of humans using their hands. Visual priors, such as
visual features, are often learned from videos, but we believe that more
information from videos can be utilized as a stronger prior. We build a
learning algorithm, VideoDex, that leverages visual, action, and physical
priors from human video datasets to guide robot behavior. These actions and
physical priors in the neural network dictate the typical human behavior for a
particular robot task. We test our approach on a robot arm and dexterous
hand-based system and show strong results on various manipulation tasks,
outperforming various state-of-the-art methods. Videos at
this https URL

利用网络视频的人类手部动作行为作为真实世界经验来指导机器人任务，使用人类视频数据集的视觉、动作、物理规律作为机器人行为的先验知识的学习算法 VideoDex，在机器人手臂和灵巧的基于手的系统上进行测试，并在各种操作任务上显示出强大的结果。