Zero-shot cross-lingual transfer is when a multilingual model is trained to
perform a task in one language and then is applied to another language.
Although the zero-shot cross-lingual transfer approach has achieved success in
various classification tasks, its performance on natural language generation
tasks falls short in quality and sometimes outputs an incorrect language. In
our study, we show that the fine-tuning process learns language invariant
representations, which is beneficial for classification tasks but harmful for
generation tasks. Motivated by this, we propose a simple method to regularize
the model from learning language invariant representations and a method to
select model checkpoints without a development set in the target language, both
resulting in better generation quality. Experiments on three semantically
diverse generation tasks show that our method reduces the accidental
translation problem by 68% and improves the ROUGE-L score by 1.5 on average.

本研究针对零样本跨语言转移中生成任务的中文输出质量不高、甚至有时会输出错误语言的问题，提出了一种用于正则化模型的简单方法和一种用于在不需要目标语言开发集的情况下选择模型检查点的方法，实验证明该方法使得 ROUGE-L 分数平均提高 1.5，同时将意外翻译问题减少了 68%。