The open-vocabulary image segmentation task involves partitioning images into
semantically meaningful segments and classifying them with flexible
text-defined categories. The recent vision-based foundation models such as the
Segment Anything Model (SAM) have shown superior performance in generating
class-agnostic image segments. The main challenge in open-vocabulary image
segmentation now lies in accurately classifying these segments into
text-defined categories. In this paper, we introduce the Universal Segment
Embedding (USE) framework to address this challenge. This framework is
comprised of two key components: 1) a data pipeline designed to efficiently
curate a large amount of segment-text pairs at various granularities, and 2) a
universal segment embedding model that enables precise segment classification
into a vast range of text-defined categories. The USE model can not only help
open-vocabulary image segmentation but also facilitate other downstream tasks
(e.g., querying and ranking). Through comprehensive experimental studies on
semantic segmentation and part segmentation benchmarks, we demonstrate that the
USE framework outperforms state-of-the-art open-vocabulary segmentation
methods.

我们介绍了通用段落嵌入（USE）框架，该框架通过数据管道和通用段落嵌入模型，能够准确地将图像划分为各种文本定义的类别，并在开放词汇图像分割和其他下游任务中取得更好的性能。

USE: 通用分段嵌入用于开放词汇库图像分割

USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation

Existing open-vocabulary image segmentation methods require a fine-tuning
step on mask annotations and/or image-text datasets. Mask labels are
labor-intensive, which limits the number of categories in segmentation
datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
under weak image-text supervision tend to make suboptimal mask predictions when
there are text queries referring to non-existing concepts in the image. To
alleviate these issues, we introduce a novel recurrent framework that
progressively filters out irrelevant texts and enhances mask quality without
training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
with frozen weights. Thus, our model retains the VLM's broad vocabulary space
and strengthens its segmentation capability. Experimental results show that our
method outperforms not only the training-free counterparts, but also those
fine-tuned with millions of additional data samples, and sets new
state-of-the-art records for both zero-shot semantic and referring image
segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

通过引入一个创新的递归框架，我们的研究表明，在不进行训练的情况下，我们的模型能够优于那些经过百万级附加样本微调的方法，为零样本语义和参考图像分割任务的最新记录设定了新的技术水平。

CLIP 作为 RNN：无需培训即可分割无限的视觉概念

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

Open-vocabulary image segmentation aims to partition an image into semantic
regions according to arbitrary text descriptions. However, complex visual
scenes can be naturally decomposed into simpler parts and abstracted at
multiple levels of granularity, introducing inherent segmentation ambiguity.
Unlike existing methods that typically sidestep this ambiguity and treat it as
an external factor, our approach actively incorporates a hierarchical
representation encompassing different semantic-levels into the learning
process. We propose a decoupled text-image fusion mechanism and representation
learning modules for both "things" and "stuff".1 Additionally, we
systematically examine the differences that exist in the textual and visual
features between these types of categories. Our resulting model, named HIPIE,
tackles HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within
a unified framework. Benchmarked on over 40 datasets, e.g., ADE20K, COCO,
Pascal-VOC Part, RefCOCO/RefCOCOg, ODinW and SeginW, HIPIE achieves the
state-of-the-art results at various levels of image comprehension, including
semantic-level (e.g., semantic segmentation), instance-level (e.g.,
panoptic/referring segmentation and object detection), as well as part-level
(e.g., part/subpart segmentation) tasks. Our code is released at
this https URL

本文提出了一种层次化结构的图像分割方法，包括分层表示和分裂的文本图像融合机制，并成功应用于多种分割任务中，包括语义、实例和部分级别的图像理解。