The pre-trained vision-language model, exemplified by CLIP, advances
zero-shot semantic segmentation by aligning visual features with class
embeddings through a transformer decoder to generate semantic masks. Despite
its effectiveness, prevailing methods within this paradigm encounter
challenges, including overfitting on seen classes and small fragmentation in
masks. To mitigate these issues, we propose a Language-Driven Visual Consensus
(LDVC) approach, fostering improved alignment of semantic and visual
information.Specifically, we leverage class embeddings as anchors due to their
discrete and abstract nature, steering vision features toward class embeddings.
Moreover, to circumvent noisy alignments from the vision part due to its
redundant nature, we introduce route attention into self-attention for finding
visual consensus, thereby enhancing semantic consistency within the same
object. Equipped with a vision-language prompting strategy, our approach
significantly boosts the generalization capacity of segmentation models for
unseen classes. Experimental results underscore the effectiveness of our
approach, showcasing mIoU gains of 4.5 on the PASCAL VOC 2012 and 3.6 on the
COCO-Stuff 164k for unseen classes compared with the state-of-the-art methods.

通过基于训练的视觉 - 语言模型，CLIP，使用变换器解码器将视觉特征与类别嵌入对齐，生成语义分割掩码，本研究提出一种语言驱动的视觉共识（LDVC）方法，通过利用类别嵌入作为锚点，引导视觉特征朝向类别嵌入，通过引入路由注意力来增强同一对象内的语义一致性，并配备视觉 - 语言提示策略，显著提升了未见类别的分割模型的泛化能力，实验结果在 PASCAL VOC 2012 和 COCO-Stuff 164k 上比现有方法在未见类别上获得了 4.5 和 3.6 的 mIoU 增益。