Vision-Language Models (VLMs) have emerged as promising tools for open-ended
image understanding tasks, including open vocabulary segmentation. Yet, direct
application of such VLMs to segmentation is non-trivial, since VLMs are trained
with image-text pairs and naturally lack pixel-level granularity. Recent works
have made advancements in bridging this gap, often by leveraging the shared
image-text space in which the image and a provided text prompt are represented.
In this paper, we challenge the capabilities of VLMs further and tackle
open-vocabulary segmentation without the need for any textual input. To this
end, we propose a novel Self-Guided Semantic Segmentation (Self-Seg) framework.
Self-Seg is capable of automatically detecting relevant class names from
clustered BLIP embeddings and using these for accurate semantic segmentation.
In addition, we propose an LLM-based Open-Vocabulary Evaluator (LOVE) to
effectively assess predicted open-vocabulary class names. We achieve
state-of-the-art results on Pascal VOC, ADE20K and CityScapes for
open-vocabulary segmentation without given class names, as well as competitive
performance with methods where class names are given. All code and data will be
released.

通过提出自主引导的语义分割（Self-Seg）框架和基于 LLM 的开放式词汇评估器（LOVE），在不需要提供类别名称的情况下，实现了开放式词汇分割的最新成果，并与提供类别名称的方法相竞争，取得了 Pascal VOC、ADE20K 和 CityScapes 的最具竞争力的结果。