Open-vocabulary instance segmentation aims at segmenting novel classes
without mask annotations. It is an important step toward reducing laborious
human supervision. Most existing works first pretrain a model on captioned
images covering many novel classes and then finetune it on limited base classes
with mask annotations. However, the high-level textual information learned from
caption pretraining alone cannot effectively encode the details required for
pixel-wise segmentation. To address this, we propose a cross-modal
pseudo-labeling framework, which generates training pseudo masks by aligning
word semantics in captions with visual features of object masks in images.
Thus, our framework is capable of labeling novel classes in captions via their
word semantics to self-train a student model. To account for noises in pseudo
masks, we design a robust student model that selectively distills mask
knowledge by estimating the mask noise levels, hence mitigating the adverse
impact of noisy pseudo masks. By extensive experiments, we show the
effectiveness of our framework, where we significantly improve mAP score by
4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions
datasets compared to the state-of-the-art.

本文提出一种跨模态伪标签（cross-modal pseudo-labeling）框架，用于面向开放词汇的实例分割，通过与对象掩膜的视觉特征对齐，实现对字词语义中的新类别进行标记，从而自我训练出一个学生模型，缓解了伪掩膜中存在的噪声干扰所带来的不良影响，相比现有工作，在 MS-COCO 和 Open Images & Conceptual Captions 数据集上分别实现了 4.5% 和 5.1% 的 mAP 得分提高。