We present an object-centric approach to empower robots to learn vision-based
manipulation skills from human videos. We investigate the problem of imitating
robot manipulation from a single human video in the open-world setting, where a
robot must learn to manipulate novel objects from one video demonstration. We
introduce ORION, an algorithm that tackles the problem by extracting an
object-centric manipulation plan from a single RGB-D video and deriving a
policy that conditions on the extracted plan. Our method enables the robot to
learn from videos captured by daily mobile devices such as an iPad and
generalize the policies to deployment environments with varying visual
backgrounds, camera angles, spatial layouts, and novel object instances. We
systematically evaluate our method on both short-horizon and long-horizon
tasks, demonstrating the efficacy of ORION in learning from a single human
video in the open world. Videos can be found in the project website
this https URL

通过从单个人类视频中提取以对象为中心的操纵计划并推导出条件为提取计划的策略，我们提出了一种以对象为中心的方法，为机器人学习基于视觉的操纵技能提供支持。我们的方法允许机器人从 iPad 等日常移动设备捕获的视频中学习，并将策略推广到具有不同视觉背景、摄像机角度、空间布局和新颖对象实例的部署环境中。通过对短程和长程任务进行系统评估，我们展示了 ORION 在从单个人类视频中学习开放世界中的有效性。

利用单个人类视频的基于视觉的开放世界物体图形操作

Vision-based Manipulation from Single Human Video with Open-World Object  Graphs

We introduce LOTUS, a continual imitation learning algorithm that empowers a
physical robot to continuously and efficiently learn to solve new manipulation
tasks throughout its lifespan. The core idea behind LOTUS is constructing an
ever-growing skill library from a sequence of new tasks with a small number of
human demonstrations. LOTUS starts with a continual skill discovery process
using an open-vocabulary vision model, which extracts skills as recurring
patterns presented in unsegmented demonstrations. Continual skill discovery
updates existing skills to avoid catastrophic forgetting of previous tasks and
adds new skills to solve novel tasks. LOTUS trains a meta-controller that
flexibly composes various skills to tackle vision-based manipulation tasks in
the lifelong learning process. Our comprehensive experiments show that LOTUS
outperforms state-of-the-art baselines by over 11% in success rate, showing its
superior knowledge transfer ability compared to prior methods. More results and
videos can be found on the project website:
this https URL

LOTUS 是一种持续模仿学习算法，使物理机器人能够在其寿命期内持续而高效地学习解决新的操作任务。通过使用一个开放词汇表视觉模型进行持续的技能发现过程，LOTUS 能构建一个不断增长的技能库，从一系列以少量人类示范展示的新任务中提取技能，并通过训练一个元控制器灵活地组合各种技能来解决基于视觉的操作任务，显示出比先前方法更强的知识传递能力。

LOTUS：无监督技能发现的机器人操控持续模仿学习

LOTUS: Continual Imitation Learning for Robot Manipulation Through  Unsupervised Skill Discovery

We introduce GROOT, an imitation learning method for learning robust policies
with object-centric and 3D priors. GROOT builds policies that generalize beyond
their initial training conditions for vision-based manipulation. It constructs
object-centric 3D representations that are robust toward background changes and
camera views and reason over these representations using a transformer-based
policy. Furthermore, we introduce a segmentation correspondence model that
allows policies to generalize to new objects at test time. Through
comprehensive experiments, we validate the robustness of GROOT policies against
perceptual variations in simulated and real-world environments. GROOT's
performance excels in generalization over background changes, camera viewpoint
shifts, and the presence of new object instances, whereas both state-of-the-art
end-to-end learning methods and object proposal-based approaches fall short. We
also extensively evaluate GROOT policies on real robots, where we demonstrate
the efficacy under very wild changes in setup. More videos and model details
can be found in the appendix and the project website:
this https URL .

GROOT 是一种基于模仿学习的方法，利用物体中心化和 3D 先验知识来学习鲁棒策略，通过构建鲁棒的物体中心化 3D 表示，并使用基于 Transformer 的策略对这些表示进行推理。此外，我们还引入了分割对应模型，使策略能够在测试时适用于新的物体。通过全面的实验，我们验证了 GROOT 在模拟和真实环境中对感知变化的鲁棒性，GROOT 在应对背景变化、摄像机视角变化和新物体实例等方面具有优越的泛化能力，而最先进的端到端学习方法和基于物体提议的方法都表现不佳。我们还在真实机器人上广泛评估了 GROOT 策略，在非常复杂的环境变化下展示了其有效性。