We build a system that enables any human to control a robot hand and arm,
simply by demonstrating motions with their own hand. The robot observes the
human operator via a single RGB camera and imitates their actions in real-time.
Human hands and robot hands differ in shape, size, and joint structure, and
performing this translation from a single uncalibrated camera is a highly
underconstrained problem. Moreover, the retargeted trajectories must
effectively execute tasks on a physical robot, which requires them to be
temporally smooth and free of self-collisions. Our key insight is that while
paired human-robot correspondence data is expensive to collect, the internet
contains a massive corpus of rich and diverse human hand videos. We leverage
this data to train a system that understands human hands and retargets a human
video stream into a robot hand-arm trajectory that is smooth, swift, safe, and
semantically similar to the guiding demonstration. We demonstrate that it
enables previously untrained people to teleoperate a robot on various dexterous
manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation
system makes robot teaching more accessible and we hope that it can aid robots
in learning to act autonomously in the real world. Videos at
this https URL

通过单个 RGB 摄像头观察人类操作者的动作并实时模仿，我们利用大规模丰富的人类手部视频数据训练了一个 system，使得未经训练的人能在各种熟练操作任务中对机器人进行远程操作，从而使机器人教学更加易于实现并可望辅助机器人在真实世界中实现自主行为。