Multilingual pre-trained language models (MPLMs) not only can handle tasks in
different languages but also exhibit surprising zero-shot cross-lingual
transferability. However, MPLMs usually are not able to achieve comparable
supervised performance on rich-resource languages compared to the
state-of-the-art monolingual pre-trained models. In this paper, we aim to
improve the multilingual model's supervised and zero-shot performance
simultaneously only with the resources from supervised languages. Our approach
is based on transferring knowledge from high-performance monolingual models
with a teacher-student framework. We let the multilingual model learn from
multiple monolingual models simultaneously. To exploit the model's
cross-lingual transferability, we propose MBLM (multi-branch multilingual
language model), a model built on the MPLMs with multiple language branches.
Each branch is a stack of transformers. MBLM is trained with the
zero-shot-aware training strategy that encourages the model to learn from the
mixture of zero-shot representations from all the branches. The results on two
cross-lingual classification tasks show that, with only the task's supervised
data used, our method improves both the supervised and zero-shot performance of
MPLMs.

利用教师 - 学生框架从高性能的单语言模型中转移知识，构建了一个基于 MPLMs 的多语言分支模型（MBLM），并使用零射击感知的训练策略令模型从所有分支的零射击表示中学习，我们的方法仅使用任务的监督数据，提高了 MPLMs 的监督性能和零射击性能。

多语言蒸馏和零样本感知训练的跨语言文本分类

Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training

We propose a simple method to align multilingual contextual embeddings as a
post-pretraining step for improved zero-shot cross-lingual transferability of
the pretrained models. Using parallel data, our method aligns embeddings on the
word level through the recently proposed Translation Language Modeling
objective as well as on the sentence level via contrastive learning and random
input shuffling. We also perform sentence-level code-switching with English
when finetuning on downstream tasks. On XNLI, our best model (initialized from
mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves
comparable result to XLM for translate-train while using less than 18% of the
same parallel data and 31% less model parameters. On MLQA, our model
outperforms XLM-R_Base that has 57% more parameters than ours.

本研究提出了一种简单的方法，作为预训练后对多语种上下文嵌入进行对齐的步骤，以提高预训练模型的零 - shot 跨语言迁移能力。该方法通过最近提出的 Translation Language Modeling 目标在词级别上对嵌入进行对齐，并通过对比学习和随机输入洗牌在句子级别上进行对齐。在下游任务的微调中，使用英语进行句子级别的代码转换。在 XNLI 上，我们的最佳模型（从 mBERT 初始化）在零 - shot 设置上比 mBERT 提高了 4.7％，在使用少于 18％的相同平行数据和 31％的模型参数的情况下，实现了与 XLM for translate-train 相当的结果。在 MLQA 上，我们的模型胜过比我们多 57％参数的 XLM-R_Base。