This paper addresses text-supervised semantic segmentation, aiming to learn a
model capable of segmenting arbitrary visual concepts within images by using
only image-text pairs without dense annotations. Existing methods have
demonstrated that contrastive learning on image-text pairs effectively aligns
visual segments with the meanings of texts. We notice that there is a
discrepancy between text alignment and semantic segmentation: A text often
consists of multiple semantic concepts, whereas semantic segmentation strives
to create semantically homogeneous segments. To address this issue, we propose
a novel framework, Image-Text Co-Decomposition (CoDe), where the paired image
and text are jointly decomposed into a set of image regions and a set of word
segments, respectively, and contrastive learning is developed to enforce
region-word alignment. To work with a vision-language model, we present a
prompt learning mechanism that derives an extra representation to highlight an
image segment or a word segment of interest, with which more effective features
can be extracted from that segment. Comprehensive experimental results
demonstrate that our method performs favorably against existing text-supervised
semantic segmentation methods on six benchmark datasets.

该论文提出了一种基于文本监督的语义分割方法，通过使用仅限于图像 - 文本对而不使用密集标注的方式学习一种能够在图像内部分割任意视觉概念的模型。通过图像 - 文本对的对比学习，有效地实现了视觉分割与文本的对齐。为了解决文本对齐与语义分割之间存在的问题，我们提出了一种新颖的框架，即图像 - 文本协同分解（CoDe），其中配对的图像和文本分别被共同分解为一组图像区域和一组单词片段，并且采用对比学习来强制执行区域 - 单词的对齐。为了配合视觉语言模型，我们提出了一种提示学习机制，从而衍生出额外的表示以突出显示感兴趣的图像段落或单词片段，从该段落中可以提取出更有效的特征。综合实验结果表明，我们的方法在六个基准数据集上优于现有的文本监督的语义分割方法。

图像文字协同分解用于基于文字监督的语义分割

Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation

Recent advancements in open vocabulary models, like CLIP, have notably
advanced zero-shot classification and segmentation by utilizing natural
language for class-specific embeddings. However, most research has focused on
improving model accuracy through prompt engineering, prompt learning, or
fine-tuning with limited labeled data, thereby overlooking the importance of
refining the class descriptors. This paper introduces a new approach to
text-supervised semantic segmentation using supervision by a large language
model (LLM) that does not require extra training. Our method starts from an
LLM, like GPT-3, to generate a detailed set of subclasses for more accurate
class representation. We then employ an advanced text-supervised semantic
segmentation model to apply the generated subclasses as target labels,
resulting in diverse segmentation results tailored to each subclass's unique
characteristics. Additionally, we propose an assembly that merges the
segmentation maps from the various subclass descriptors to ensure a more
comprehensive representation of the different aspects in the test images.
Through comprehensive experiments on three standard benchmarks, our method
outperforms traditional text-supervised semantic segmentation methods by a
marked margin.

该论文介绍了一种利用大型语言模型（LLM）进行文本监督语义分割的新方法，该方法通过生成更准确的类别表示来提供多样化的分割结果，并通过合并不同的子类描述符的分割图确保对测试图像的更全面的表示。经过三个标准基准的全面实验，我们的方法比传统的文本监督语义分割方法表现出更好的性能。

基于 LLM 监督的零训练语义分割

Training-Free Semantic Segmentation via LLM-Supervision

Recently, great success has been made in learning visual representations from
text supervision, facilitating the emergence of text-supervised semantic
segmentation. However, existing works focus on pixel grouping and cross-modal
semantic alignment, while ignoring the correspondence among multiple augmented
views of the same image. To overcome such limitation, we propose
multi-\textbf{View} \textbf{Co}nsistent learning (ViewCo) for text-supervised
semantic segmentation. Specifically, we first propose text-to-views consistency
modeling to learn correspondence for multiple views of the same input image.
Additionally, we propose cross-view segmentation consistency modeling to
address the ambiguity issue of text supervision by contrasting the segment
features of Siamese visual encoders. The text-to-views consistency benefits the
dense assignment of the visual features by encouraging different crops to align
with the same text, while the cross-view segmentation consistency modeling
provides additional self-supervision, overcoming the limitation of ambiguous
text supervision for segmentation masks. Trained with large-scale image-text
data, our model can directly segment objects of arbitrary categories in a
zero-shot manner. Extensive experiments show that ViewCo outperforms
state-of-the-art methods on average by up to 2.9\%, 1.6\%, and 2.4\% mIoU on
PASCAL VOC2012, PASCAL Context, and COCO, respectively.

本文提出了一种基于文本监督的分割方法，使用多视角一致性学习的方法解决了现有工作中对于同一图像的多个视角的对应问题。实验结果表明，该方法在几个数据集上的平均表现比现有方法提高了最高 2.9％的 mIoU。