We propose a simple method to align multilingual contextual embeddings as a
post-pretraining step for improved zero-shot cross-lingual transferability of
the pretrained models. Using parallel data, our method aligns embeddings on the
word level through the recently proposed Translation Language Modeling
objective as well as on the sentence level via contrastive learning and random
input shuffling. We also perform sentence-level code-switching with English
when finetuning on downstream tasks. On XNLI, our best model (initialized from
mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves
comparable result to XLM for translate-train while using less than 18% of the
same parallel data and 31% less model parameters. On MLQA, our model
outperforms XLM-R_Base that has 57% more parameters than ours.

本研究提出了一种简单的方法，作为预训练后对多语种上下文嵌入进行对齐的步骤，以提高预训练模型的零 - shot 跨语言迁移能力。该方法通过最近提出的 Translation Language Modeling 目标在词级别上对嵌入进行对齐，并通过对比学习和随机输入洗牌在句子级别上进行对齐。在下游任务的微调中，使用英语进行句子级别的代码转换。在 XNLI 上，我们的最佳模型（从 mBERT 初始化）在零 - shot 设置上比 mBERT 提高了 4.7％，在使用少于 18％的相同平行数据和 31％的模型参数的情况下，实现了与 XLM for translate-train 相当的结果。在 MLQA 上，我们的模型胜过比我们多 57％参数的 XLM-R_Base。

多语言 BERT 模型预训练后对齐

Multilingual BERT Post-Pretraining Alignment

While BERT is an effective method for learning monolingual sentence
embeddings for semantic similarity and embedding based transfer learning
(Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have
yet to be explored. We systematically investigate methods for learning
multilingual sentence embeddings by combining the best methods for learning
monolingual and cross-lingual representations including: masked language
modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019),
dual encoder translation ranking (Guo et al., 2018), and additive margin
softmax (Yang et al., 2019a). We show that introducing a pre-trained
multilingual language model dramatically reduces the amount of parallel
training data required to achieve good performance by 80%. Composing the best
of these methods produces a model that achieves 83.7% bi-text retrieval
accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by
Artetxe and Schwenk (2019b), while still performing competitively on
monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel
data mined from CommonCrawl using our best model is shown to train competitive
NMT models for en-zh and en-de. We publicly release our best multilingual
sentence embedding model for 109+ languages at this https URL

本研究旨在通过探究多种单语和跨语言表示学习方法，如掩码语言建模，翻译语言建模和双编码器翻译排名等，结合预训练的多语言模型来学习多语言句子嵌入，并成功将其用于多语言文本检索和机器翻译任务中。