Cross-lingual Cross-modal Retrieval (CCR) is an essential task in web search,
which aims to break the barriers between modality and language simultaneously
and achieves image-text retrieval in the multi-lingual scenario with a single
model. In recent years, excellent progress has been made based on cross-lingual
cross-modal pre-training; particularly, the methods based on contrastive
learning on large-scale data have significantly improved retrieval tasks.
However, these methods directly follow the existing pre-training methods in the
cross-lingual or cross-modal domain, leading to two problems of inconsistency
in CCR: The methods with cross-lingual style suffer from the intra-modal error
propagation, resulting in inconsistent recall performance across languages in
the whole dataset. The methods with cross-modal style suffer from the
inter-modal optimization direction bias, resulting in inconsistent rank across
languages within each instance, which cannot be reflected by Recall@K. To solve
these problems, we propose a simple but effective 1-to-K contrastive learning
method, which treats each language equally and eliminates error propagation and
optimization bias. In addition, we propose a new evaluation metric, Mean Rank
Variance (MRV), to reflect the rank inconsistency across languages within each
instance. Extensive experiments on four CCR datasets show that our method
improves both recall rates and MRV with smaller-scale pre-trained data,
achieving the new state-of-art.

提出了一种简单但有效的 1 对 K 对比学习方法，平等对待每种语言，消除错误传播和优化偏差的问题，并提出了一个新的评估指标 Mean Rank Variance（MRV），来反映每个实例内不同语言之间的排名不一致。在四个 CCR 数据集上进行的广泛实验表明，我们的方法在小规模预训练数据上提高了召回率和 MRV，并取得了新的最新成果。

提升跨语言跨模态检索中的一致性：基于 1-to-K 对比学习

Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with  1-to-K Contrastive Learning

Cross-lingual cross-modal retrieval has garnered increasing attention
recently, which aims to achieve the alignment between vision and target
language (V-T) without using any annotated V-T data pairs. Current methods
employ machine translation (MT) to construct pseudo-parallel data pairs, which
are then used to learn a multi-lingual and multi-modal embedding space that
aligns visual and target-language representations. However, the large
heterogeneous gap between vision and text, along with the noise present in
target language translations, poses significant challenges in effectively
aligning their representations. To address these challenges, we propose a
general framework, Cross-Lingual to Cross-Modal (CL2CM), which improves the
alignment between vision and target language using cross-lingual transfer. This
approach allows us to fully leverage the merits of multi-lingual pre-trained
models (e.g., mBERT) and the benefits of the same modality structure, i.e.,
smaller gap, to provide reliable and comprehensive semantic correspondence
(knowledge) for the cross-modal network. We evaluate our proposed approach on
two multilingual image-text datasets, Multi30K and MSCOCO, and one video-text
dataset, VATEX. The results clearly demonstrate the effectiveness of our
proposed method and its high potential for large-scale retrieval.

跨语言跨模态检索致力于在不使用任何标注的视觉 - 目标语言数据对的情况下实现视觉和目标语言之间的对齐。本研究提出了一种名为 CL2CM 的通用框架，使用跨语言转移改善了视觉和目标语言之间的对齐，可在跨模态网络中为可靠全面的语义对应（知识）提供优势，并通过在多语言图像 - 文本数据集和视频 - 文本数据集上进行实验验证了其高潜力和有效性。