Teleoperation serves as a powerful method for collecting on-robot data
essential for robot learning from demonstrations. The intuitiveness and ease of
use of the teleoperation system are crucial for ensuring high-quality, diverse,
and scalable data. To achieve this, we propose an immersive teleoperation
system Open-TeleVision that allows operators to actively perceive the robot's
surroundings in a stereoscopic manner. Additionally, the system mirrors the
operator's arm and hand movements on the robot, creating an immersive
experience as if the operator's mind is transmitted to a robot embodiment. We
validate the effectiveness of our system by collecting data and training
imitation learning policies on four long-horizon, precise tasks (Can Sorting,
Can Insertion, Folding, and Unloading) for 2 different humanoid robots and
deploy them in the real world. The system is open-sourced at:
this https URL

我们提出了一种沉浸式远程操作系统 Open-TeleVision，用于收集机器人学习演示所必需的在机器人上的数据，通过在一个立体的方式中使操作者主动感知机器人的周围环境，并将操作者的手臂和手部动作映射到机器人上，创造出一种沉浸式的体验，我们通过在真实环境中收集数据并训练模仿学习策略，验证了我们系统的有效性。

Open-TeleVision: 具有沉浸式主动视觉反馈的远程操作

Open-TeleVision: Teleoperation with Immersive Active Visual Feedback

In this paper we leverage self-supervised vision transformer models and their
emergent semantic abilities to improve the generalization abilities of
imitation learning policies. We introduce BC-ViT, an imitation learning
algorithm that leverages rich DINO pre-trained Visual Transformer (ViT)
patch-level embeddings to obtain better generalization when learning through
demonstrations. Our learner sees the world by clustering appearance features
into semantic concepts, forming stable keypoints that generalize across a wide
range of appearance variations and object types. We show that this
representation enables generalized behaviour by evaluating imitation learning
across a diverse dataset of object manipulation tasks. Our method, data and
evaluation approach are made available to facilitate further study of
generalization in Imitation Learners.

利用自我监督的视觉变换模型及其新出的语义能力，通过聚类外观特征来形成稳定的关键点，从而改善模仿学习策略的泛化能力。本论文介绍了 BC-ViT，一种利用富含 DINO 预训练视觉变换器（ViT）补丁级嵌入的模仿学习算法，以通过示范获取更好的泛化效果。通过对一个多样化的物体操作任务数据集进行模仿学习的评估，证明了这种表示方式能够实现广义行为。为了促进对于模仿学习中泛化问题的进一步研究，我们提供了我们的方法、数据和评估方法。