Pre-trained multilingual language models such as mBERT have shown immense
gains for several natural language processing (NLP) tasks, especially in the
zero-shot cross-lingual setting. Most, if not all, of these pre-trained models
rely on the masked-language modeling (MLM) objective as the key language
learning objective. The principle behind these approaches is that predicting
the masked words with the help of the surrounding text helps learn potent
contextualized representations. Despite the strong representation learning
capability enabled by MLM, we demonstrate an inherent limitation of MLM for
multilingual representation learning. In particular, by requiring the model to
predict the language-specific token, the MLM objective disincentivizes learning
a language-agnostic representation -- which is a key goal of multilingual
pre-training. Therefore to encourage better cross-lingual representation
learning we propose the DICT-MLM method. DICT-MLM works by incentivizing the
model to be able to predict not just the original masked word, but potentially
any of its cross-lingual synonyms as well. Our empirical analysis on multiple
downstream tasks spanning 30+ languages, demonstrates the efficacy of the
proposed approach and its ability to learn better multilingual representations.

本文提出 DICT-MLM 方法以促进跨语言表示学习，分析表明该方法在多种涉及 30 多种语言的下游任务中表现出了更好的多语言表示能力。