In this paper, we study a challenging task of zero-shot referring image
segmentation. This task aims to identify the instance mask that is most related
to a referring expression without training on pixel-level annotations. Previous
research takes advantage of pre-trained cross-modal models, e.g., CLIP, to
align instance-level masks with referring expressions. %Yet, CLIP only
considers image-text pair level alignment, which neglects fine-grained image
region and complex sentence matching. Yet, CLIP only considers the global-level
alignment of image-text pairs, neglecting fine-grained matching between the
referring sentence and local image regions. To address this challenge, we
introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image
segmentation framework that is training-free and robust to various visual
encoders. TAS incorporates a mask proposal network for instance-level mask
extraction, a text-augmented visual-text matching score for mining the
image-text correlation, and a spatial rectifier for mask post-processing.
Notably, the text-augmented visual-text matching score leverages a $P$ score
and an $N$-score in addition to the typical visual-text matching score. The
$P$-score is utilized to close the visual-text domain gap through a surrogate
captioning model, where the score is computed between the surrogate
model-generated texts and the referring expression. The $N$-score considers the
fine-grained alignment of region-text pairs via negative phrase mining,
encouraging the masked image to be repelled from the mined distracting phrases.
Extensive experiments are conducted on various datasets, including RefCOCO,
RefCOCO+, and RefCOCOg. The proposed method clearly outperforms
state-of-the-art zero-shot referring image segmentation methods.

通过引入具有文本增强的空间感知（TAS）零样本指代图像分割框架，本研究解决了零样本指代图像分割任务中的挑战，该任务旨在在没有像素级注释的情况下识别与指代表达最相关的实例掩码，并且在不同的视觉编码器上具有训练免费和鲁棒性的特点。

文本增强的空间感知零样本指代图像分割

Text Augmented Spatial-aware Zero-shot Referring Image Segmentation

Zero-shot referring image segmentation is a challenging task because it aims
to find an instance segmentation mask based on the given referring
descriptions, without training on this type of paired data. Current zero-shot
methods mainly focus on using pre-trained discriminative models (e.g., CLIP).
However, we have observed that generative models (e.g., Stable Diffusion) have
potentially understood the relationships between various visual elements and
text descriptions, which are rarely investigated in this task. In this work, we
introduce a novel Referring Diffusional segmentor (Ref-Diff) for this task,
which leverages the fine-grained multi-modal information from generative
models. We demonstrate that without a proposal generator, a generative model
alone can achieve comparable performance to existing SOTA weakly-supervised
models. When we combine both generative and discriminative models, our Ref-Diff
outperforms these competing methods by a significant margin. This indicates
that generative models are also beneficial for this task and can complement
discriminative models for better referring segmentation. Our code is publicly
available at this https URL

本研究提出了一种用于零样本参照图像分割的新型方法 Ref-Diff，它利用生成模型中的细粒度多模态信息，证明仅凭生成模型的性能可以与现有的 SOTA 弱监督模型相媲美，并且当将生成模型与判别模型结合时，我们的 Ref-Diff 显著优于其他方法，证明生成模型对于该任务也是有益的，可为更好的参照分割提供补充。