We focus on the task of unknown object rearrangement, where a robot is
supposed to re-configure the objects into a desired goal configuration
specified by an RGB-D image. Recent works explore unknown object rearrangement
systems by incorporating learning-based perception modules. However, they are
sensitive to perception error, and pay less attention to task-level
performance. In this paper, we aim to develop an effective system for unknown
object rearrangement amidst perception noise. We theoretically reveal the noisy
perception impacts grasp and place in a decoupled way, and show such a
decoupled structure is non-trivial to improve task optimality. We propose GSP,
a dual-loop system with the decoupled structure as prior. For the inner loop,
we learn an active seeing policy for self-confident object matching to improve
the perception of place. For the outer loop, we learn a grasp policy aware of
object matching and grasp capability guided by task-level rewards. We leverage
the foundation model CLIP for object matching, policy learning and
self-termination. A series of experiments indicate that GSP can conduct unknown
object rearrangement with higher completion rate and less steps.

我们提出 GSP，一个双环系统，以解决在感知噪声中进行未知物体重新排列的问题。通过学习的感知模块进行对象匹配和抓取能力的任务导向的奖励指导，GSP 在完成率和步骤数方面表现更好。