An outstanding image-text retrieval model depends on high-quality labeled
data. While the builders of existing image-text retrieval datasets strive to
ensure that the caption matches the linked image, they cannot prevent a caption
from fitting other images. We observe that such a many-to-many matching
phenomenon is quite common in the widely-used retrieval datasets, where one
caption can describe up to 178 images. These large matching-lost data not only
confuse the model in training but also weaken the evaluation accuracy. Inspired
by visual and textual entailment tasks, we propose a multi-modal entailment
classifier to determine whether a sentence is entailed by an image plus its
linked captions. Subsequently, we revise the image-text retrieval datasets by
adding these entailed captions as additional weak labels of an image and
develop a universal variable learning rate strategy to teach a retrieval model
to distinguish the entailed captions from other negative samples. In
experiments, we manually annotate an entailment-corrected image-text retrieval
dataset for evaluation. The results demonstrate that the proposed entailment
classifier achieves about 78% accuracy and consistently improves the
performance of image-text retrieval baselines.

本论文提出了一种多模态蕴含分类器来确定图像的语句，以及开发了一种通用可调节学习速率策略来教授一个检索模型区分这些蕴含的语句和其他负样本。在实验中，我们手动注释了一个蕴含更正的图像文本检索数据集用于评估。结果表明，所提出的蕴含分类器达到了约 78% 的准确度，并持续提高了图像文本检索基线的性能。