Recent studies have demonstrated the potential of cross-lingual
transferability by training a unified Transformer encoder for multiple
languages. In addition to involving the masked language model objective,
existing cross-lingual pre-training works leverage sentence-level contrastive
learning or plugs in extra cross-attention module to complement the
insufficient capabilities of cross-lingual alignment. Nonetheless, synonym
pairs residing in bilingual corpus are not exploited and aligned, which is more
crucial than sentence interdependence establishment for token-level tasks. In
this work, we propose a cross-lingual pre-trained model VECO~2.0 based on
contrastive learning with multi-granularity alignments. Specifically, the
sequence-to-sequence alignment is induced to maximize the similarity of the
parallel pairs and minimize the non-parallel pairs. Then, token-to-token
alignment is integrated to bridge the gap between synonymous tokens excavated
via the thesaurus dictionary from the other unpaired tokens in a bilingual
instance. Experiments show the effectiveness of the proposed strategy for
cross-lingual model pre-training on the XTREME benchmark.

本文提出了一种基于对比学习的多粒度对齐的跨语言预训练模型 VECO~2.0，将序列到序列对齐和标记到标记对齐相结合，以支持跨语言模型预训练，通过在 XTREME 基准上的实验证明了该方法的有效性。