While large-scale robotic systems typically rely on textual instructions for
tasks, this work explores a different approach: can robots infer the task
directly from observing humans? This shift necessitates the robot's ability to
decode human intent and translate it into executable actions within its
physical constraints and environment. We introduce Vid2Robot, a novel
end-to-end video-based learning framework for robots. Given a video
demonstration of a manipulation task and current visual observations, Vid2Robot
directly produces robot actions. This is achieved through a unified
representation model trained on a large dataset of human video and robot
trajectory. The model leverages cross-attention mechanisms to fuse prompt video
features to the robot's current state and generate appropriate actions that
mimic the observed task. To further improve policy performance, we propose
auxiliary contrastive losses that enhance the alignment between human and robot
video representations. We evaluate Vid2Robot on real-world robots,
demonstrating a 20% improvement in performance compared to other
video-conditioned policies when using human demonstration videos. Additionally,
our model exhibits emergent capabilities, such as successfully transferring
observed motions from one object to another, and long-horizon composition, thus
showcasing its potential for real-world applications. Project website:
vid2robot.github.io

通过观察人类行为并将其翻译成可执行的动作，本研究介绍了一种基于视频学习的机器人框架 Vid2Robot，它通过训练机器人模型利用人类视频和机器人轨迹数据集进行任务执行。该模型利用交叉注意力机制将提示视频特征融合到机器人的当前状态中，并生成能够模仿所观察任务的适当动作，大幅提升执行效果，同时显示着在真实世界应用中的潜力。

Vid2Robot: 基于跨引注意力变形器的端到端视频条件策略学习

Vid2Robot: End-to-end Video-conditioned Policy Learning with  Cross-Attention Transformers

Human demonstration videos are a widely available data source for robot
learning and an intuitive user interface for expressing desired behavior.
However, directly extracting reusable robot manipulation skills from
unstructured human videos is challenging due to the big embodiment difference
and unobserved action parameters. To bridge this embodiment gap, this paper
introduces XSkill, an imitation learning framework that 1) discovers a
cross-embodiment representation called skill prototypes purely from unlabeled
human and robot manipulation videos, 2) transfers the skill representation to
robot actions using conditional diffusion policy, and finally, 3) composes the
learned skill to accomplish unseen tasks specified by a human prompt video. Our
experiments in simulation and real-world environments show that the discovered
skill prototypes facilitate both skill transfer and composition for unseen
tasks, resulting in a more general and scalable imitation learning framework.
The performance of XSkill is best understood from the anonymous website:
this https URL

在这篇论文中，介绍了一种名为 XSkill 的模仿学习框架，通过从未标记的人类和机器人操纵视频中发现一种跨体现性表示，使用条件扩散策略将该表示转移到机器人行为，并通过组合学习到的技能完成由人类提示视频指定的未见任务，进而解决了直接从非结构化人类视频中提取可重用机器人操作技能的挑战。实验结果表明，XSkill 的性能更好。

XSkill：跨体现技能发现

XSkill: Cross Embodiment Skill Discovery

We study the problem of generalizable task learning from human demonstration
videos without extra training on the robot or pre-recorded robot motions. Given
a set of human demonstration videos showing a task with different objects/tools
(categorical objects), we aim to learn a representation of visual observation
that generalizes to categorical objects and enables efficient controller
design. We propose to introduce a geometric task structure to the
representation learning problem that geometrically encodes the task
specification from human demonstration videos, and that enables generalization
by building task specification correspondence between categorical objects.
Specifically, we propose CoVGS-IL, which uses a graph-structured task function
to learn task representations under structural constraints. Our method enables
task generalization by selecting geometric features from different objects
whose inner connection relationships define the same task in geometric
constraints. The learned task representation is then transferred to a robot
controller using uncalibrated visual servoing (UVS); thus, the need for extra
robot training or pre-recorded robot motions is removed.

本文提出了一种利用几何任务结构进行表示学习的方法，通过人类演示视频学习视觉观察的表征，并通过建立不同类别物体之间的任务说明对应来实现任务的泛化，从而不需要额外的机器人训练或预先录制的机器人运动来实现在机器人上的任务学习。