As one of the most fundamental techniques in multimodal learning, cross-modal
matching aims to project various sensory modalities into a shared feature
space. To achieve this, massive and correctly aligned data pairs are required
for model training. However, unlike unimodal datasets, multimodal datasets are
extremely harder to collect and annotate precisely. As an alternative, the
co-occurred data pairs (e.g., image-text pairs) collected from the Internet
have been widely exploited in the area. Unfortunately, the cheaply collected
dataset unavoidably contains many mismatched data pairs, which have been proven
to be harmful to the model's performance. To address this, we propose a general
framework called BiCro (Bidirectional Cross-modal similarity consistency),
which can be easily integrated into existing cross-modal matching models and
improve their robustness against noisy data. Specifically, BiCro aims to
estimate soft labels for noisy data pairs to reflect their true correspondence
degree. The basic idea of BiCro is motivated by that -- taking image-text
matching as an example -- similar images should have similar textual
descriptions and vice versa. Then the consistency of these two similarities can
be recast as the estimated soft labels to train the matching model. The
experiments on three popular cross-modal matching datasets demonstrate that our
method significantly improves the noise-robustness of various matching models,
and surpass the state-of-the-art by a clear margin.

该研究提出了一种名为 BiCro 的框架，可提高交叉模态匹配模型的噪声鲁棒性，通过估计噪声数据对应度的软标签来训练匹配模型。实验结果表明，该方法明显提高了各种匹配模型的抗噪声能力，并领先于同类研究成果。