Pre-trained multilingual language models (LMs) have achieved state-of-the-art
results in cross-lingual transfer, but they often lead to an inequitable
representation of languages due to limited capacity, skewed pre-training data,
and sub-optimal vocabularies. This has prompted the creation of an ever-growing
pre-trained model universe, where each model is trained on large amounts of
language or domain specific data with a carefully curated, linguistically
informed vocabulary. However, doing so brings us back full circle and prevents
one from leveraging the benefits of multilinguality. To address the gaps at
both ends of the spectrum, we propose MergeDistill, a framework to merge
pre-trained LMs in a way that can best leverage their assets with minimal
dependencies, using task-agnostic knowledge distillation. We demonstrate the
applicability of our framework in a practical setting by leveraging
pre-existing teacher LMs and training student LMs that perform competitively
with or even outperform teacher LMs trained on several orders of magnitude more
data and with a fixed model capacity. We also highlight the importance of
teacher selection and its impact on student model performance.

本文提出了 MergeDistill 框架，通过知识蒸馏的方式合并预先训练的多语言模型，旨在快速训练出性能与甚至优于数量级更大的模型，同时强调教师模型的选择对学生模型性能的影响。