Referring expression comprehension (REC) and segmentation (RES) are two
highly-related tasks, which both aim at identifying the referent according to a
natural language expression. In this paper, we propose a novel Multi-task
Collaborative Network (MCN) to achieve a joint learning of REC and RES for the
first time. In MCN, RES can help REC to achieve better language-vision
alignment, while REC can help RES to better locate the referent. In addition,
we address a key challenge in this multi-task setup, i.e., the prediction
conflict, with two innovative designs namely, Consistency Energy Maximization
(CEM) and Adaptive Soft Non-Located Suppression (ASNLS). Specifically, CEM
enables REC and RES to focus on similar visual regions by maximizing the
consistency energy between two tasks. ASNLS supresses the response of unrelated
regions in RES based on the prediction of REC. To validate our model, we
conduct extensive experiments on three benchmark datasets of REC and RES, i.e.,
RefCOCO, RefCOCO+ and RefCOCOg. The experimental results report the significant
performance gains of MCN over all existing methods, i.e., up to +7.13% for REC
and +11.50% for RES over SOTA, which well confirm the validity of our model for
joint REC and RES learning.

本研究提出一种新的多任务协作网络 (MCN) 模型通过联合学习来实现指代表达理解 (REC) 和分割 (RES) 两个高度相关的任务，并通过创新设计解决预测冲突的挑战，实验结果表明，在三个基准数据集 RefCOCO、RefCOCO+ 和 RefCOCOg 上，MCN 模型相较于现有方法有了显著的表现提升，REC 和 RES 的准确率分别提高了 7.13% 和 11.50%，证实了我们的方法的有效性。