Accurate alignment between languages is fundamental for improving
cross-lingual pre-trained language models (XLMs). Motivated by the natural
phenomenon of code-switching (CS) in multilingual speakers, CS has been used as
an effective data augmentation method that offers language alignment at the
word- or phrase-level, in contrast to sentence-level via parallel instances.
Existing approaches either use dictionaries or parallel sentences with word
alignment to generate CS data by randomly switching words in a sentence.
However, such methods can be suboptimal as dictionaries disregard semantics,
and syntax might become invalid after random word switching. In this work, we
propose EntityCS, a method that focuses on Entity-level Code-Switching to
capture fine-grained cross-lingual semantics without corrupting syntax. We use
Wikidata and English Wikipedia to construct an entity-centric CS corpus by
switching entities to their counterparts in other languages. We further propose
entity-oriented masking strategies during intermediate model training on the
EntityCS corpus for improving entity prediction. Evaluation of the trained
models on four entity-centric downstream tasks shows consistent improvements
over the baseline with a notable increase of 10% in Fact Retrieval. We release
the corpus and models to assist research on code-switching and enriching XLMs
with external knowledge.

本文提出了一种实体级别的语言混合方法（EntityCS），不仅可以避免语法错误，还可以提高四个实体中心下游任务的性能，在 Fact Retrieval 任务上最高可达 10%。

EntityCS：基于实体为中心的代码切换改进零样本跨语言迁移

EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric  Code Switching

Recent studies have demonstrated the efficiency of generative pretraining for
English natural language understanding. In this work, we extend this approach
to multiple languages and show the effectiveness of cross-lingual pretraining.
We propose two methods to learn cross-lingual language models (XLMs): one
unsupervised that only relies on monolingual data, and one supervised that
leverages parallel data with a new cross-lingual language model objective. We
obtain state-of-the-art results on cross-lingual classification, unsupervised
and supervised machine translation. On XNLI, our approach pushes the state of
the art by an absolute gain of 4.9% accuracy. On unsupervised machine
translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the
previous state of the art by more than 9 BLEU. On supervised machine
translation, we obtain a new state of the art of 38.5 BLEU on WMT'16
Romanian-English, outperforming the previous best approach by more than 4 BLEU.
Our code and pretrained models will be made publicly available.

本文提出了两种跨语言学习模型的方法 (XLMs): 一种是仅依赖于单语数据的无监督方式， 另一种是利用新的跨语言模型目标并使用平行数据的有监督方式。通过这些方法在跨语言分类、无监督和有监督机器翻译中取得了最先进的结果。