Common image-text joint understanding techniques presume that images and the
associated text can universally be characterized by a single implicit model.
However, co-occurring images and text can be related in qualitatively different
ways, and explicitly modeling it could improve the performance of current joint
understanding models. In this paper, we train a Cross-Modal Coherence Modelfor
text-to-image retrieval task. Our analysis shows that models trained with
image--text coherence relations can retrieve images originally paired with
target text more often than coherence-agnostic models. We also show via human
evaluation that images retrieved by the proposed coherence-aware model are
preferred over a coherence-agnostic baseline by a huge margin. Our findings
provide insights into the ways that different modalities communicate and the
role of coherence relations in capturing commonsense inferences in text and
imagery.

本文提出了一种跨模态相依的一致性模型，用于提高现有联合理解模型的性能，结果展示出使用相依性理念的模型能够更好地检索与目标文本相关的图像，这对不同模态间的沟通及在捕获文本与图像常识推理中相依性关系的角色具有重要的研究意义。