Semantic segmentation has a broad range of applications, but its real-world
impact has been significantly limited by the prohibitive annotation costs
necessary to enable deployment. Segmentation methods that forgo supervision can
side-step these costs, but exhibit the inconvenient requirement to provide
labelled examples from the target distribution to assign concept names to
predictions. An alternative line of work in language-image pre-training has
recently demonstrated the potential to produce models that can both assign
names across large vocabularies of concepts and enable zero-shot transfer for
classification, but do not demonstrate commensurate segmentation abilities. In
this work, we strive to achieve a synthesis of these two approaches that
combines their strengths. We leverage the retrieval abilities of one such
language-image pre-trained model, CLIP, to dynamically curate training sets
from unlabelled images for arbitrary collections of concept names, and leverage
the robust correspondences offered by modern image representations to
co-segment entities among the resulting collections. The synthetic segment
collections are then employed to construct a segmentation model (without
requiring pixel labels) whose knowledge of concepts is inherited from the
scalable pre-training process of CLIP. We demonstrate that our approach, termed
Retrieve and Co-segment (ReCo) performs favourably to unsupervised segmentation
approaches while inheriting the convenience of nameable predictions and
zero-shot transfer. We also demonstrate ReCo's ability to generate specialist
segmenters for extremely rare objects.

本文提出了一种将语言 - 图像预训练模型（如 CLIP）的检索能力与无监督图像分割方法结合的方法，称为 Retrieve and Co-segment（ReCo）。该方法可构建语义分割模型，无需像素级别的标注，具有词汇表中概念名称的预测方便性和零样本转移能力，并能生成少见对象的专用分割模型。