Existing semantic segmentation approaches are often limited by costly
pixel-wise annotations and predefined classes. In this work, we present
CLIP-S$^4$ that leverages self-supervised pixel representation learning and
vision-language models to enable various semantic segmentation tasks (e.g.,
unsupervised, transfer learning, language-driven segmentation) without any
human annotations and unknown class information. We first learn pixel
embeddings with pixel-segment contrastive learning from different augmented
views of images. To further improve the pixel embeddings and enable
language-driven semantic segmentation, we design two types of consistency
guided by vision-language models: 1) embedding consistency, aligning our pixel
embeddings to the joint feature space of a pre-trained vision-language model,
CLIP; and 2) semantic consistency, forcing our model to make the same
predictions as CLIP over a set of carefully designed target classes with both
known and unknown prototypes. Thus, CLIP-S$^4$ enables a new task of class-free
semantic segmentation where no unknown class information is needed during
training. As a result, our approach shows consistent and substantial
performance improvement over four popular benchmarks compared with the
state-of-the-art unsupervised and language-driven semantic segmentation
methods. More importantly, our method outperforms these methods on unknown
class recognition by a large margin.

本文通过自监督学习及视觉 - 语言模型，提出了 CLIP-S4 方法，该方法可以在不需要人类注释和未知类信息的情况下进行各种语义分割任务，包括无监督、迁移学习和语言驱动分割，并在未知类别识别上表现出良好的性能优势。