In recent years, pre-trained multilingual language models, such as multilingual BERT and XLM-R, exhibit good performance on zero-shot cross-lingual transfer learning. However, since their multilingual contextual embedding spaces for different languages are not perfectly aligned, the difference between representations of different languages might cause zero-shot cross-lingual transfer failed in some cases. In this work, we draw connections between those failed cases and adversarial examples. We then propose to use robust training methods to train a robust model that can tolerate some noise in input embeddings. We study two widely used robust training methods: adversarial training and randomized smoothing. The experimental results demonstrate that robust training can improve zero-shot cross-lingual transfer for text classification. The performance improvements become significant when the distance between the source language and the target language increases.

本文提出了一种通过对抗样本和零样本跨语言转移失败案例进行联系的学习策略，采用对抗性训练和随机平滑这两种方法来训练多语言编码器更加强健的模型，实验结果表明，强健训练可以提高零样本跨语言数据分类任务中的性能，尤其在输入语句属于两种不同语言的情况下，改进更为显著。

通过鲁棒性训练提升零样本跨语言迁移学习