In this paper, we study a challenging task of zero-shot referring image
segmentation. This task aims to identify the instance mask that is most related
to a referring expression without training on pixel-level annotations. Previous
research takes advantage of pre-trained cross-modal models, e.g., CLIP, to
align instance-level masks with referring expressions. %Yet, CLIP only
considers image-text pair level alignment, which neglects fine-grained image
region and complex sentence matching. Yet, CLIP only considers the global-level
alignment of image-text pairs, neglecting fine-grained matching between the
referring sentence and local image regions. To address this challenge, we
introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image
segmentation framework that is training-free and robust to various visual
encoders. TAS incorporates a mask proposal network for instance-level mask
extraction, a text-augmented visual-text matching score for mining the
image-text correlation, and a spatial rectifier for mask post-processing.
Notably, the text-augmented visual-text matching score leverages a $P$ score
and an $N$-score in addition to the typical visual-text matching score. The
$P$-score is utilized to close the visual-text domain gap through a surrogate
captioning model, where the score is computed between the surrogate
model-generated texts and the referring expression. The $N$-score considers the
fine-grained alignment of region-text pairs via negative phrase mining,
encouraging the masked image to be repelled from the mined distracting phrases.
Extensive experiments are conducted on various datasets, including RefCOCO,
RefCOCO+, and RefCOCOg. The proposed method clearly outperforms
state-of-the-art zero-shot referring image segmentation methods.

通过引入具有文本增强的空间感知（TAS）零样本指代图像分割框架，本研究解决了零样本指代图像分割任务中的挑战，该任务旨在在没有像素级注释的情况下识别与指代表达最相关的实例掩码，并且在不同的视觉编码器上具有训练免费和鲁棒性的特点。