Zero-shot classification capabilities naturally arise in models trained
within a vision-language contrastive framework. Despite their classification
prowess, these models struggle in dense tasks like zero-shot open-vocabulary
segmentation. This deficiency is often attributed to the absence of
localization cues in captions and the intertwined nature of the learning
process, which encompasses both image representation learning and
cross-modality alignment. To tackle these issues, we propose SimZSS, a Simple
framework for open-vocabulary Zero-Shot Segmentation. The method is founded on
two key principles: i) leveraging frozen vision-only models that exhibit
spatial awareness while exclusively aligning the text encoder and ii)
exploiting the discrete nature of text and linguistic knowledge to pinpoint
local concepts within captions. By capitalizing on the quality of the visual
representations, our method requires only image-caption pairs datasets and
adapts to both small curated and large-scale noisy datasets. When trained on
COCO Captions across 8 GPUs, SimZSS achieves state-of-the-art results on 7 out
of 8 benchmark datasets in less than 15 minutes.

通过使用冻结的仅包含视觉信息的模型和融合离散文本和语言知识的方式，我们提出了 SimZSS，一种用于开放性词汇的零样本分割的简单框架。在仅使用图像 - 标题对数据集进行训练的情况下，我们的方法利用了视觉表示的质量，并在不到 15 分钟的时间内在 8 个基准数据集中的 7 个上实现了最先进的结果。