Multilingual pre-trained models exhibit zero-shot cross-lingual transfer,
where a model fine-tuned on a source language achieves surprisingly good
performance on a target language. While studies have attempted to understand
transfer, they focus only on MLM, and the large number of differences between
natural languages makes it hard to disentangle the importance of different
properties. In this work, we specifically highlight the importance of word
embedding alignment by proposing a pre-training objective (ALIGN-MLM) whose
auxiliary loss guides similar words in different languages to have similar word
embeddings. ALIGN-MLM either outperforms or matches three widely adopted
objectives (MLM, XLM, DICT-MLM) when we evaluate transfer between pairs of
natural languages and their counterparts created by systematically modifying
specific properties like the script. In particular, ALIGN-MLM outperforms XLM
and MLM by 35 and 30 F1 points on POS-tagging for transfer between languages
that differ both in their script and word order (left-to-right v.s.
right-to-left). We also show a strong correlation between alignment and
transfer for all objectives (e.g., rho=0.727 for XNLI), which together with
ALIGN-MLM's strong performance calls for explicitly aligning word embeddings
for multilingual models.

本研究提出了一种新的预训练目标 ALIGN-MLM，该目标通过辅助损失来引导不同语言中相似的词具有相似的词嵌入，实现跨语言转移学习中的单词嵌入对齐，此方法在机器学习中的应用具有广泛的前景。

ALIGN-MLM: 多语言预训练中的词嵌入对齐至关重要

ALIGN-MLM: Word Embedding Alignment is Crucial for Multilingual Pre-training

While recent work on multilingual language models has demonstrated their
capacity for cross-lingual zero-shot transfer on downstream tasks, there is a
lack of consensus in the community as to what shared properties between
languages enable such transfer. Analyses involving pairs of natural languages
are often inconclusive and contradictory since languages simultaneously differ
in many linguistic aspects. In this paper, we perform a large-scale empirical
study to isolate the effects of various linguistic properties by measuring
zero-shot transfer between four diverse natural languages and their
counterparts constructed by modifying aspects such as the script, word order,
and syntax. Among other things, our experiments show that the absence of
sub-word overlap significantly affects zero-shot transfer when languages differ
in their word order, and there is a strong correlation between transfer
performance and word embedding alignment between languages (e.g., R=0.94 on the
task of NLI). Our results call for focus in multilingual models on explicitly
improving word embedding alignment between languages rather than relying on its
implicit emergence.

该论文在多语言语言模型上进行了大规模实证研究，并发现词嵌入的对齐程度与零 - shot 迁移的性能密切相关，因此需要在多语言模型中专门改善词嵌入的对齐程度。