In this paper, we study multimodal coreference resolution, specifically where
a longer descriptive text, i.e., a narration is paired with an image. This
poses significant challenges due to fine-grained image-text alignment, inherent
ambiguity present in narrative language, and unavailability of large annotated
training sets. To tackle these challenges, we present a data efficient
semi-supervised approach that utilizes image-narration pairs to resolve
coreferences and narrative grounding in a multimodal context. Our approach
incorporates losses for both labeled and unlabeled data within a cross-modal
framework. Our evaluation shows that the proposed approach outperforms strong
baselines both quantitatively and qualitatively, for the tasks of coreference
resolution and narrative grounding.

研究采用图像和描述性文本的多模态共指消解，在细粒度的图像 - 文本对齐、叙述语言的内在歧义和缺乏大规模标注数据集的条件下，提出了一种数据效率的半监督方法，用于解决多模态背景下的共指消解和叙述依托问题，通过跨模态框架结合有标注和无标注数据的损失优化，实验证明该方法在共指消解和叙述依托任务上的性能优于强基线模型。