Referring image segmentation is a typical multi-modal task, which aims at
generating a binary mask for referent described in given language expressions.
Prior arts adopt a bimodal solution, taking images and languages as two
modalities within an encoder-fusion-decoder pipeline. However, this pipeline is
sub-optimal for the target task for two reasons. First, they only fuse
high-level features produced by uni-modal encoders separately, which hinders
sufficient cross-modal learning. Second, the uni-modal encoders are pre-trained
independently, which brings inconsistency between pre-trained uni-modal tasks
and the target multi-modal task. Besides, this pipeline often ignores or makes
little use of intuitively beneficial instance-level features. To relieve these
problems, we propose MaIL, which is a more concise encoder-decoder pipeline
with a Mask-Image-Language trimodal encoder. Specifically, MaIL unifies
uni-modal feature extractors and their fusion model into a deep modality
interaction encoder, facilitating sufficient feature interaction across
different modalities. Meanwhile, MaIL directly avoids the second limitation
since no uni-modal encoders are needed anymore. Moreover, for the first time,
we propose to introduce instance masks as an additional modality, which
explicitly intensifies instance-level features and promotes finer segmentation
results. The proposed MaIL set a new state-of-the-art on all frequently-used
referring image segmentation datasets, including RefCOCO, RefCOCO+, and G-Ref,
with significant gains, 3%-10% against previous best methods. Code will be
released soon.

提出 MaIL 范式，是一个更加简洁的编码器 - 解码器流程，通过遮罩图像语言三模态编码器统一单模态特征提取器和它们的融合模型，实现不同模态间的充分特征交互，同时避免了单模态编码器与目标多模态任务之间的一致性问题，并首次在所有常用的引用图像分割数据集上引入实例掩模作为附加模态，显式强调实例级特征，并促进更细致的分割结果。