The ability for robots to comprehend and execute manipulation tasks based on
natural language instructions is a long-term goal in robotics. The dominant
approaches for language-guided manipulation use 2D image representations, which
face difficulties in combining multi-view cameras and inferring precise 3D
positions and relationships. To address these limitations, we propose a 3D
point cloud based policy called PolarNet for language-guided manipulation. It
leverages carefully designed point cloud inputs, efficient point cloud
encoders, and multimodal transformers to learn 3D point cloud representations
and integrate them with language instructions for action prediction. PolarNet
is shown to be effective and data efficient in a variety of experiments
conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D
approaches in both single-task and multi-task learning. It also achieves
promising results on a real robot.

通过使用 3D 点云来进行语言引导的操纵任务，我们提出了一种名为 PolarNet 的策略，该策略通过精心设计的点云输入、高效的点云编码器和多模态变换器来学习 3D 点云表示并将其与语言指令集成以进行行动预测。在 RLBench 基准测试中，PolarNet 在各种实验中显示出了高效和数据高效性，并在单任务和多任务学习中优于最先进的 2D 和 3D 方法。它在实际机器人上也取得了令人期待的结果。

PolarNet: 基于语言引导的机器人操作的 3D 点云

PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation

The need for fine-grained perception in autonomous driving systems has
resulted in recently increased research on online semantic segmentation of
single-scan LiDAR. Despite the emerging datasets and technological
advancements, it remains challenging due to three reasons: (1) the need for
near-real-time latency with limited hardware; (2) uneven or even long-tailed
distribution of LiDAR points across space; and (3) an increasing number of
extremely fine-grained semantic classes. In an attempt to jointly tackle all
the aforementioned challenges, we propose a new LiDAR-specific,
nearest-neighbor-free segmentation algorithm - PolarNet. Instead of using
common spherical or bird's-eye-view projection, our polar bird's-eye-view
representation balances the points across grid cells in a polar coordinate
system, indirectly aligning a segmentation network's attention with the
long-tailed distribution of the points along the radial axis. We find that our
encoding scheme greatly increases the mIoU in three drastically different
segmentation datasets of real urban LiDAR single scans while retaining near
real-time throughput.

本文提出了针对 LiDAR 数据的新型自动驾驶车辆的语义分割算法 PolarNet，通过極坐标系间接对齐分割网络的注意力，解决了点云分布不均等问题。在三个不同的真实城市 LiDAR 单次扫描数据集中，该编码方案显著提高了平均交并比，同时保持了接近实时的吞吐量。