The ability for robots to comprehend and execute manipulation tasks based on
natural language instructions is a long-term goal in robotics. The dominant
approaches for language-guided manipulation use 2D image representations, which
face difficulties in combining multi-view cameras and inferring precise 3D
positions and relationships. To address these limitations, we propose a 3D
point cloud based policy called PolarNet for language-guided manipulation. It
leverages carefully designed point cloud inputs, efficient point cloud
encoders, and multimodal transformers to learn 3D point cloud representations
and integrate them with language instructions for action prediction. PolarNet
is shown to be effective and data efficient in a variety of experiments
conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D
approaches in both single-task and multi-task learning. It also achieves
promising results on a real robot.

通过使用 3D 点云来进行语言引导的操纵任务，我们提出了一种名为 PolarNet 的策略，该策略通过精心设计的点云输入、高效的点云编码器和多模态变换器来学习 3D 点云表示并将其与语言指令集成以进行行动预测。在 RLBench 基准测试中，PolarNet 在各种实验中显示出了高效和数据高效性，并在单任务和多任务学习中优于最先进的 2D 和 3D 方法。它在实际机器人上也取得了令人期待的结果。

PolarNet: 基于语言引导的机器人操作的 3D 点云

PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation

Benefiting from language flexibility and compositionality, humans naturally
intend to use language to command an embodied agent for complex tasks such as
navigation and object manipulation. In this work, we aim to fill the blank of
the last mile of embodied agents -- object manipulation by following human
guidance, e.g., "move the red mug next to the box while keeping it upright." To
this end, we introduce an Automatic Manipulation Solver (AMSolver) system and
build a Vision-and-Language Manipulation benchmark (VLMbench) based on it,
containing various language instructions on categorized robotic manipulation
tasks. Specifically, modular rule-based task templates are created to
automatically generate robot demonstrations with language instructions,
consisting of diverse object shapes and appearances, action types, and motion
constraints. We also develop a keypoint-based model 6D-CLIPort to deal with
multi-view observations and language input and output a sequence of 6 degrees
of freedom (DoF) actions. We hope the new simulator and benchmark will
facilitate future research on language-guided robotic manipulation.

本文提出自动操作求解器（AMSolver）系统和基于其构建的视觉与语言操作基准（VLMbench），用于处理基于语言指令的机器人操作任务，并开发了基于关键点的 6D-CLIPort 模型来处理多视角观察和语言输入并输出一系列 6 自由度（DoF）动作。