Pre-trained multilingual language models underpin a large portion of modern
NLP tools outside of English. A strong baseline for specializing these models
for specific languages is Language-Adaptive Pre-Training (LAPT). However,
retaining a large cross-lingual vocabulary and embedding matrix comes at
considerable excess computational cost during adaptation. In this study, we
propose several simple techniques to replace a cross-lingual vocabulary with a
compact, language-specific one. Namely, we address strategies for
re-initializing the token embedding matrix after vocabulary specialization. We
then provide a systematic experimental comparison of our techniques, in
addition to the recently-proposed Focus method. We demonstrate that: 1)
Embedding-replacement techniques in the monolingual transfer literature are
inadequate for adapting multilingual models. 2) Replacing cross-lingual
vocabularies with smaller specialized ones provides an efficient method to
improve performance in low-resource languages. 3) Simple embedding
re-initialization techniques based on script-wise sub-distributions rival
techniques such as Focus, which rely on similarity scores obtained from an
auxiliary model.

本研究提出了几种简单的技术，通过替换跨语言词汇为紧凑的特定语言词汇来改善在低资源语言中的性能，以及基于脚本子分布的嵌入重初始化技术在适应多语言模型方面与依赖于辅助模型获得的相似度分数的技术（如焦点方法）相媲美。