Instance segmentation with unseen objects is a challenging problem in unstructured environments. To solve this problem, we propose a robot learning approach to actively interact with novel objects and collect each object's training label for further fine-tuning to improve the segmentation model performance, while avoiding the time-consuming process of manually labeling a dataset. The Singulation-and-Grasping (SaG) policy is trained through end-to-end reinforcement learning. Given a cluttered pile of objects, our approach chooses pushing and grasping motions to break the clutter and conducts object-agnostic grasping for which the SaG policy takes as input the visual observations and imperfect segmentation. We decompose the problem into three subtasks: (1) the object singulation subtask aims to separate the objects from each other, which creates more space that alleviates the difficulty of (2) the collision-free grasping subtask; (3) the mask generation subtask to obtain the self-labeled ground truth masks by using an optical flow-based binary classifier and motion cue post-processing for transfer learning. Our system achieves 70% singulation success rate in simulated cluttered scenes. The interactive segmentation of our system achieves 87.8%, 73.9%, and 69.3% average precision for toy blocks, YCB objects in simulation and real-world novel objects, respectively, which outperforms several baselines.

该研究提出了一种机器人学习方法，利用在未见过新物体时的交互作用，与避免手动标记数据集的费时过程，收集每个物体的训练标签进行微调来提高分段模型的性能。该系统实现了在模拟杂乱场景中 70% 的分离成功率，而对于玩具积木、模拟 YCB 对象和真实世界的新物体，交互式分割的平均精度分别达到了 87.8%、73.9% 和 69.3%，超过了几个基线。

基于分离和抓取方法的自监督交互式物体分割