For multilingual sequence-to-sequence pretrained language models
(multilingual Seq2Seq PLMs), e.g. mBART, the self-supervised pretraining task
is trained on a wide range of monolingual languages, e.g. 25 languages from
CommonCrawl, while the downstream cross-lingual tasks generally progress on a
bilingual language subset, e.g. English-German, making there exists the data
discrepancy, namely domain discrepancy, and cross-lingual learning objective
discrepancy, namely task discrepancy, between the pretraining and finetuning
stages. To bridge the above cross-lingual domain and task gaps, we extend the
vanilla pretrain-finetune pipeline with extra code-switching restore task.
Specifically, the first stage employs the self-supervised code-switching
restore task as a pretext task, allowing the multilingual Seq2Seq PLMs to
acquire some in-domain alignment information. And for the second stage, we
fine-tune the model on downstream data normally. Experiments on both NLG
evaluation (12 bilingual translation tasks, 30 zero-shot translation tasks, and
2 cross-lingual summarization tasks) and NLU evaluation (7 cross-lingual
natural language inference tasks) show our model outperforms the strong
baseline mBART with standard finetuning strategy, consistently. Analyses
indicate our approach could narrow the Euclidean distance of cross-lingual
sentence representations, and improve the model generalization with trivial
computational cost. We release the code at:
this https URL.

本文提出了一种通过引入代码切换任务来减少跨语言场景下的数据和任务差异从而提高多语言序列到序列预训练语言模型（multilingual Seq2Seq PLMs) 的性能的方法，实验结果表明该方法显著优于以标准微调策略微调的基准模型 mBART，可以缩小跨语言句子表示的欧几里得距离，并且在计算成本方面没有太多的增加。