We present an approach for building an active agent that learns to segment its visual observations into individual objects by interacting with its environment in a completely self-supervised manner. The agent uses its current segmentation model to infer pixels that constitute objects and refines the segmentation model by interacting with these pixels. The model learned from over 50K interactions generalizes to novel objects and backgrounds. To deal with noisy training signal for segmenting objects obtained by self-supervised interactions, we propose robust set loss. A dataset of robot's interactions along-with a few human labeled examples is provided as a benchmark for future research. We test the utility of the learned segmentation model by providing results on a downstream vision-based control task of rearranging multiple objects into target configurations from visual inputs alone. Videos, code, and robotic interaction dataset are available at https://pathak22.github.io/seg-by-interaction/

本研究提出了一种利用自我监督方式以及与环境互动的方法，建立一个学习将视觉观察分割成不同对象的主动代理。通过对50,000次以上与物体交互的学习，该代理的分割模型能够推广到新的物体和背景，并通过提供视频、代码以及与机器人的互动数据集来测试所学分割模型的效用。

通过交互学习实例分割