Cross-lingual Cross-modal Retrieval (CCR) is an essential task in web search,
which aims to break the barriers between modality and language simultaneously
and achieves image-text retrieval in the multi-lingual scenario with a single
model. In recent years, excellent progress has been made based on cross-lingual
cross-modal pre-training; particularly, the methods based on contrastive
learning on large-scale data have significantly improved retrieval tasks.
However, these methods directly follow the existing pre-training methods in the
cross-lingual or cross-modal domain, leading to two problems of inconsistency
in CCR: The methods with cross-lingual style suffer from the intra-modal error
propagation, resulting in inconsistent recall performance across languages in
the whole dataset. The methods with cross-modal style suffer from the
inter-modal optimization direction bias, resulting in inconsistent rank across
languages within each instance, which cannot be reflected by Recall@K. To solve
these problems, we propose a simple but effective 1-to-K contrastive learning
method, which treats each language equally and eliminates error propagation and
optimization bias. In addition, we propose a new evaluation metric, Mean Rank
Variance (MRV), to reflect the rank inconsistency across languages within each
instance. Extensive experiments on four CCR datasets show that our method
improves both recall rates and MRV with smaller-scale pre-trained data,
achieving the new state-of-art.

提出了一种简单但有效的 1 对 K 对比学习方法，平等对待每种语言，消除错误传播和优化偏差的问题，并提出了一个新的评估指标 Mean Rank Variance（MRV），来反映每个实例内不同语言之间的排名不一致。在四个 CCR 数据集上进行的广泛实验表明，我们的方法在小规模预训练数据上提高了召回率和 MRV，并取得了新的最新成果。