In this paper, we study multimodal coreference resolution, specifically where
a longer descriptive text, i.e., a narration is paired with an image. This
poses significant challenges due to fine-grained image-text alignment, inherent
ambiguity present in narrative language, and unavailability of large annotated
training sets. To tackle these challenges, we present a data efficient
semi-supervised approach that utilizes image-narration pairs to resolve
coreferences and narrative grounding in a multimodal context. Our approach
incorporates losses for both labeled and unlabeled data within a cross-modal
framework. Our evaluation shows that the proposed approach outperforms strong
baselines both quantitatively and qualitatively, for the tasks of coreference
resolution and narrative grounding.

研究采用图像和描述性文本的多模态共指消解，在细粒度的图像 - 文本对齐、叙述语言的内在歧义和缺乏大规模标注数据集的条件下，提出了一种数据效率的半监督方法，用于解决多模态背景下的共指消解和叙述依托问题，通过跨模态框架结合有标注和无标注数据的损失优化，实验证明该方法在共指消解和叙述依托任务上的性能优于强基线模型。

图像叙述中的半监督多模态指代消解

Semi-supervised multimodal coreference resolution in image narrations

We apply a generative segmental model of task structure, guided by narration,
to action segmentation in video. We focus on unsupervised and weakly-supervised
settings where no action labels are known during training. Despite its
simplicity, our model performs competitively with previous work on a dataset of
naturalistic instructional videos. Our model allows us to vary the sources of
supervision used in training, and we find that both task structure and
narrative language provide large benefits in segmentation quality.

应用叙述指导下的生成式分段模型，重点研究在无监督和弱监督设置中进行视频动作分割，发现任务结构和叙述语言在训练中被用作监督来源有利于提高分割质量.