Zero-shot referring image segmentation is a challenging task because it aims
to find an instance segmentation mask based on the given referring
descriptions, without training on this type of paired data. Current zero-shot
methods mainly focus on using pre-trained discriminative models (e.g., CLIP).
However, we have observed that generative models (e.g., Stable Diffusion) have
potentially understood the relationships between various visual elements and
text descriptions, which are rarely investigated in this task. In this work, we
introduce a novel Referring Diffusional segmentor (Ref-Diff) for this task,
which leverages the fine-grained multi-modal information from generative
models. We demonstrate that without a proposal generator, a generative model
alone can achieve comparable performance to existing SOTA weakly-supervised
models. When we combine both generative and discriminative models, our Ref-Diff
outperforms these competing methods by a significant margin. This indicates
that generative models are also beneficial for this task and can complement
discriminative models for better referring segmentation. Our code is publicly
available at this https URL

本研究提出了一种用于零样本参照图像分割的新型方法 Ref-Diff，它利用生成模型中的细粒度多模态信息，证明仅凭生成模型的性能可以与现有的 SOTA 弱监督模型相媲美，并且当将生成模型与判别模型结合时，我们的 Ref-Diff 显著优于其他方法，证明生成模型对于该任务也是有益的，可为更好的参照分割提供补充。