Pre-trained multilingual language encoders, such as multilingual BERT and
XLM-R, show great potential for zero-shot cross-lingual transfer. However,
these multilingual encoders do not precisely align words and phrases across
languages. Especially, learning alignments in the multilingual embedding space
usually requires sentence-level or word-level parallel corpora, which are
expensive to be obtained for low-resource languages. An alternative is to make
the multilingual encoders more robust; when fine-tuning the encoder using
downstream task, we train the encoder to tolerate noise in the contextual
embedding spaces such that even if the representations of different languages
are not aligned well, the model can still achieve good performance on zero-shot
cross-lingual transfer. In this work, we propose a learning strategy for
training robust models by drawing connections between adversarial examples and
the failure cases of zero-shot cross-lingual transfer. We adopt two widely used
robust training methods, adversarial training and randomized smoothing, to
train the desired robust model. The experimental results demonstrate that
robust training improves zero-shot cross-lingual transfer on text
classification tasks. The improvement is more significant in the generalized
cross-lingual transfer setting, where the pair of input sentences belong to two
different languages.

本文提出了一种通过对抗样本和零样本跨语言转移失败案例进行联系的学习策略，采用对抗性训练和随机平滑这两种方法来训练多语言编码器更加强健的模型，实验结果表明，强健训练可以提高零样本跨语言数据分类任务中的性能，尤其在输入语句属于两种不同语言的情况下，改进更为显著。