We design an open-vocabulary image segmentation model to organize an image
into meaningful regions indicated by arbitrary texts. Recent works (CLIP and
ALIGN), despite attaining impressive open-vocabulary classification accuracy
with image-level caption labels, are unable to segment visual concepts with
pixels. We argue that these models miss an important step of visual grouping,
which organizes pixels into groups before learning visual-semantic alignments.
We propose OpenSeg to address the above issue while still making use of
scalable image-level supervision of captions. First, it learns to propose
segmentation masks for possible organizations. Then it learns visual-semantic
alignments by aligning each word in a caption to one or a few predicted masks.
We find the mask representations are the key to support learning image
segmentation from captions, making it possible to scale up the dataset and
vocabulary sizes. OpenSeg significantly outperforms the recent open-vocabulary
method of LSeg by +19.9 mIoU on PASCAL dataset, thanks to its scalability.

本文提出了 OpenSeg 模型，通过学习像素的可视化分组和视觉 - 语义对齐，利用可扩展的图像级监督生成分割掩码，从而实现对图像的开放词汇的分割与组织，相对 LSeg 模型在 PASCAL 数据集上精度提高了 19.9 个百分点。