The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-labels word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rates on the alignment benchmarks. The code and pretrained parameters are available at https://github.com/CZWin32768/XLM-Align.

该研究探讨了使用去噪词对齐作为新的跨语言预训练任务的方法，通过在双语对中进行自我标注的单词对齐，结合指针网络来预测清理片段中被屏蔽的单词及其在另一种语言中的对应单词，并在期望值最大化的方式下交替进行两个步骤，实验结果表明该方法在各种数据集上提高了跨语言可传递性。

通过自标注的词对齐改进预训练的跨语言语言模型