Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT),
have shown success in a variety of zero-shot cross-lingual tasks. However,
these models are limited by having inconsistent contextualized representations
of subwords across different languages. Existing work addresses this issue by
bilingual projection and fine-tuning technique. We propose a data augmentation
framework to generate multi-lingual code-switching data to fine-tune mBERT,
which encourages model to align representations from source and multiple target
languages once by mixing their context information. Compared with the existing
work, our method does not rely on bilingual sentences for training, and
requires only one training process for multiple target languages. Experimental
results on five tasks with 19 languages show that our method leads to
significantly improved performances for all the tasks compared with mBERT.

提出了一种数据增强框架以生成多语言混合数据来微调多语言 - BERT 模型，从而实现将源语言和多个目标语言的表示进行对齐，相较现有方法，该方法无需依赖双语句子进行训练，并且只需一个训练过程即可对多个目标语言进行微调，对于 19 种语言的五项任务表现均明显提高。