Current research on cross-modal retrieval is mostly English-oriented, as the
availability of a large number of English-oriented human-labeled
vision-language corpora. In order to break the limit of non-English labeled
data, cross-lingual cross-modal retrieval (CCR) has attracted increasing
attention. Most CCR methods construct pseudo-parallel vision-language corpora
via Machine Translation (MT) to achieve cross-lingual transfer. However, the
translated sentences from MT are generally imperfect in describing the
corresponding visual contents. Improperly assuming the pseudo-parallel data are
correctly correlated will make the networks overfit to the noisy
correspondence. Therefore, we propose Dual-view Curricular Optimal Transport
(DCOT) to learn with noisy correspondence in CCR. In particular, we quantify
the confidence of the sample pair correlation with optimal transport theory
from both the cross-lingual and cross-modal views, and design dual-view
curriculum learning to dynamically model the transportation costs according to
the learning stage of the two views. Extensive experiments are conducted on two
multilingual image-text datasets and one video-text dataset, and the results
demonstrate the effectiveness and robustness of the proposed method. Besides,
our proposed method also shows a good expansibility to cross-lingual image-text
baselines and a decent generalization on out-of-domain data.

通过 Dual-view Curricular Optimal Transport（DCOT）在跨语言跨模态检索中学习带有噪声的对应关系，从而解决机器翻译产生的不完美翻译问题，实验证明了该方法的有效性和鲁棒性。