Referring Image Segmentation (RIS) aims to segment an object described in
natural language from an image, with the main challenge being a text-to-pixel
correlation. Previous methods typically rely on single-modality features, such
as vision or language features, to guide the multi-modal fusion process.
However, this approach limits the interaction between vision and language,
leading to a lack of fine-grained correlation between the language description
and pixel-level details during the decoding process. In this paper, we
introduce FCNet, a framework that employs a bi-directional guided fusion
approach where both vision and language play guiding roles. Specifically, we
use a vision-guided approach to conduct initial multi-modal fusion, obtaining
multi-modal features that focus on key vision information. We then propose a
language-guided calibration module to further calibrate these multi-modal
features, ensuring they understand the context of the input sentence. This
bi-directional vision-language guided approach produces higher-quality
multi-modal features sent to the decoder, facilitating adaptive propagation of
fine-grained semantic information from textual features to visual features.
Experiments on RefCOCO, RefCOCO+, and G-Ref datasets with various backbones
consistently show our approach outperforming state-of-the-art methods.

本文介绍了 FCNet（一个使用双向引导融合方法的框架），其中视觉和语言在引导角色中相互作用，旨在解决自然语言描述与像素级细节之间的关联问题，并通过在多模态特征上进行初始融合和进一步校准，提高多模态特征的质量。实验结果表明，我们的方法在多个数据集上优于现有最先进算法。