For 3D object manipulation, methods that build an explicit 3D representation
perform better than those relying only on camera images. But using explicit 3D
representations like voxels comes at large computing cost, adversely affecting
scalability. In this work, we propose RVT, a multi-view transformer for 3D
manipulation that is both scalable and accurate. Some key features of RVT are
an attention mechanism to aggregate information across views and re-rendering
of the camera input from virtual views around the robot workspace. In
simulations, we find that a single RVT model works well across 18 RLBench tasks
with 249 task variations, achieving 26% higher relative success than the
existing state-of-the-art method (PerAct). It also trains 36X faster than
PerAct for achieving the same performance and achieves 2.3X the inference speed
of PerAct. Further, RVT can perform a variety of manipulation tasks in the real
world with just a few ($\sim$10) demonstrations per task. Visual results, code,
and trained model are provided at this https URL

本研究提出 RVT，一种基于多视角变形器的三维物体操作方法，在模拟和实际环境中均表现出色，相对于现有方法，其训练速度快 36 倍，推理速度快 2.3 倍，并仅需每项任务约 10 次演示即可达到良好的效果。

RVT: 用于 3D 对象操作的机器人视图变换器

RVT: Robotic View Transformer for 3D Object Manipulation

The 3D visual grounding task aims to ground a natural language description to
the targeted object in a 3D scene, which is usually represented in 3D point
clouds. Previous works studied visual grounding under specific views. The
vision-language correspondence learned by this way can easily fail once the
view changes. In this paper, we propose a Multi-View Transformer (MVT) for 3D
visual grounding. We project the 3D scene to a multi-view space, in which the
position information of the 3D scene under different views are modeled
simultaneously and aggregated together. The multi-view space enables the
network to learn a more robust multi-modal representation for 3D visual
grounding and eliminates the dependence on specific views. Extensive
experiments show that our approach significantly outperforms all
state-of-the-art methods. Specifically, on Nr3D and Sr3D datasets, our method
outperforms the best competitor by 11.2% and 7.1% and even surpasses recent
work with extra 2D assistance by 5.9% and 6.6%. Our code is available at
this https URL

这篇论文提出了一种基于多视角变换器（MVT）的方法，将 3D 场景投影到多视角空间中，学习一种更为稳健的多模态表示形式，从而消除特定视角的依赖，并在实验中表现出比现有方法更好的性能。