Robots rely heavily on sensors, especially RGB and depth cameras, to perceive
and interact with the world. RGB cameras record 2D images with rich semantic
information while missing precise spatial information. On the other side, depth
cameras offer critical 3D geometry data but capture limited semantics.
Therefore, integrating both modalities is crucial for learning representations
for robotic perception and control. However, current research predominantly
focuses on only one of these modalities, neglecting the benefits of
incorporating both. To this end, we present Semantic-Geometric Representation
(SGR), a universal perception module for robotics that leverages the rich
semantic information of large-scale pre-trained 2D models and inherits the
merits of 3D spatial reasoning. Our experiments demonstrate that SGR empowers
the agent to successfully complete a diverse range of simulated and real-world
robotic manipulation tasks, outperforming state-of-the-art methods
significantly in both single-task and multi-task settings. Furthermore, SGR
possesses the unique capability to generalize to novel semantic attributes,
setting it apart from the other methods.

本文提出一种名为语义几何表征（SGR）的感知模块，旨在整合 RGB 和深度相机的信息，以更好地学习机器人感知和控制的表征，实验结果表明，SGR 在单任务和多任务设置下的表现均显著优于其他方法，并具有到新语义属性的泛化能力。