Referring image segmentation aims to segment the target object referred by a
natural language expression. However, previous methods rely on the strong
assumption that one sentence must describe one target in the image, which is
often not the case in real-world applications. As a result, such methods fail
when the expressions refer to either no objects or multiple objects. In this
paper, we address this issue from two perspectives. First, we propose a Dual
Multi-Modal Interaction (DMMI) Network, which contains two decoder branches and
enables information flow in two directions. In the text-to-image decoder, text
embedding is utilized to query the visual feature and localize the
corresponding target. Meanwhile, the image-to-text decoder is implemented to
reconstruct the erased entity-phrase conditioned on the visual feature. In this
way, visual features are encouraged to contain the critical semantic
information about target entity, which supports the accurate segmentation in
the text-to-image decoder in turn. Secondly, we collect a new challenging but
realistic dataset called Ref-ZOM, which includes image-text pairs under
different settings. Extensive experiments demonstrate our method achieves
state-of-the-art performance on different datasets, and the Ref-ZOM-trained
model performs well on various types of text inputs. Codes and datasets are
available at this https URL

本文提出了一种双重多模态交互网络 (DMMI)，用于解决自然语言表达中引用图像分割的问题，并设计了一个新的挑战性但真实的数据集 Ref-ZOM，通过大量实验证明该方法在不同数据集上取得了最先进的性能，并且 Ref-ZOM 训练的模型能够良好地处理各种类型的文本输入。