Despite the significant progress in deep learning for dense visual
recognition problems, such as semantic segmentation, traditional methods are
constrained by fixed class sets. Meanwhile, vision-language foundation models,
such as CLIP, have showcased remarkable effectiveness in numerous zero-shot
image-level tasks, owing to their robust generalizability. Recently, a body of
work has investigated utilizing these models in open-vocabulary semantic
segmentation (OVSS). However, existing approaches often rely on impractical
supervised pre-training or access to additional pre-trained networks. In this
work, we propose a strong baseline for training-free OVSS, termed
Neighbour-Aware CLIP (NACLIP), representing a straightforward adaptation of
CLIP tailored for this scenario. Our method enforces localization of patches in
the self-attention of CLIP's vision transformer which, despite being crucial
for dense prediction tasks, has been overlooked in the OVSS literature. By
incorporating design choices favouring segmentation, our approach significantly
improves performance without requiring additional data, auxiliary pre-trained
networks, or extensive hyperparameter tuning, making it highly practical for
real-world applications. Experiments are performed on 8 popular semantic
segmentation benchmarks, yielding state-of-the-art performance on most
scenarios. Our code is publicly available at this https URL .

借鉴了 CLIP 模型，并通过自适应 CLIP 的可视化 transformer 来强化感知，提出了一种新的方法 NACLIP 用于训练自由的 open-vocabulary semantic segmentation (OVSS)，在 8 个流行的语义分割评估中获得了最先进的性能。