While large-scale robotic systems typically rely on textual instructions for
tasks, this work explores a different approach: can robots infer the task
directly from observing humans? This shift necessitates the robot's ability to
decode human intent and translate it into executable actions within its
physical constraints and environment. We introduce Vid2Robot, a novel
end-to-end video-based learning framework for robots. Given a video
demonstration of a manipulation task and current visual observations, Vid2Robot
directly produces robot actions. This is achieved through a unified
representation model trained on a large dataset of human video and robot
trajectory. The model leverages cross-attention mechanisms to fuse prompt video
features to the robot's current state and generate appropriate actions that
mimic the observed task. To further improve policy performance, we propose
auxiliary contrastive losses that enhance the alignment between human and robot
video representations. We evaluate Vid2Robot on real-world robots,
demonstrating a 20% improvement in performance compared to other
video-conditioned policies when using human demonstration videos. Additionally,
our model exhibits emergent capabilities, such as successfully transferring
observed motions from one object to another, and long-horizon composition, thus
showcasing its potential for real-world applications. Project website:
vid2robot.github.io

通过观察人类行为并将其翻译成可执行的动作，本研究介绍了一种基于视频学习的机器人框架 Vid2Robot，它通过训练机器人模型利用人类视频和机器人轨迹数据集进行任务执行。该模型利用交叉注意力机制将提示视频特征融合到机器人的当前状态中，并生成能够模仿所观察任务的适当动作，大幅提升执行效果，同时显示着在真实世界应用中的潜力。