Vision-Language Models (VLMs) have emerged as promising tools for open-ended
image understanding tasks, including open vocabulary segmentation. Yet, direct
application of such VLMs to segmentation is non-trivial, since VLMs are trained
with image-text pairs and naturally lack pixel-level granularity. Recent works
have made advancements in bridging this gap, often by leveraging the shared
image-text space in which the image and a provided text prompt are represented.
In this paper, we challenge the capabilities of VLMs further and tackle
open-vocabulary segmentation without the need for any textual input. To this
end, we propose a novel Self-Guided Semantic Segmentation (Self-Seg) framework.
Self-Seg is capable of automatically detecting relevant class names from
clustered BLIP embeddings and using these for accurate semantic segmentation.
In addition, we propose an LLM-based Open-Vocabulary Evaluator (LOVE) to
effectively assess predicted open-vocabulary class names. We achieve
state-of-the-art results on Pascal VOC, ADE20K and CityScapes for
open-vocabulary segmentation without given class names, as well as competitive
performance with methods where class names are given. All code and data will be
released.

通过提出自主引导的语义分割（Self-Seg）框架和基于 LLM 的开放式词汇评估器（LOVE），在不需要提供类别名称的情况下，实现了开放式词汇分割的最新成果，并与提供类别名称的方法相竞争，取得了 Pascal VOC、ADE20K 和 CityScapes 的最具竞争力的结果。

自导引开放词汇语义分割

Self-Guided Open-Vocabulary Semantic Segmentation

Few-shot learning (FSL) is the task of learning to recognize previously
unseen categories of images from a small number of training examples. This is a
challenging task, as the available examples may not be enough to unambiguously
determine which visual features are most characteristic of the considered
categories. To alleviate this issue, we propose a method that additionally
takes into account the names of the image classes. While the use of class names
has already been explored in previous work, our approach differs in two key
aspects. First, while previous work has aimed to directly predict visual
prototypes from word embeddings, we found that better results can be obtained
by treating visual and text-based prototypes separately. Second, we propose a
simple strategy for learning class name embeddings using the BERT language
model, which we found to substantially outperform the GloVe vectors that were
used in previous work. We furthermore propose a strategy for dealing with the
high dimensionality of these vectors, inspired by models for aligning
cross-lingual word embeddings. We provide experiments on miniImageNet, CUB and
tieredImageNet, showing that our approach consistently improves the
state-of-the-art in metric-based FSL.

提出了一种考虑图像类别名称的少样本学习方法，利用 BERT 模型学习类别名称嵌入来分离视觉和文本特征，同时采用跨语言词向量对高维度 Bert 嵌入进行处理，实验结果表明该方法在少样本图像分类任务上取得了领先水平。