Compared to monolingual models, cross-lingual models usually require a more
expressive vocabulary to represent all languages adequately. We find that many
languages are under-represented in recent cross-lingual language models due to
the limited vocabulary capacity. To this end, we propose an algorithm VoCap to
determine the desired vocabulary capacity of each language. However, increasing
the vocabulary size significantly slows down the pre-training speed. In order
to address the issues, we propose k-NN-based target sampling to accelerate the
expensive softmax. Our experiments show that the multilingual vocabulary
learned with VoCap benefits cross-lingual language model pre-training.
Moreover, k-NN-based target sampling mitigates the side-effects of increasing
the vocabulary size while achieving comparable performance and faster
pre-training speed. The code and the pretrained multilingual vocabularies are
available at this https URL

通过 VoCap 算法确定语言所需的词汇容量并使用 k-NN 目标采样来加速昂贵的 softmax，从而缓解词汇量增加的副作用，提高交叉语言模型的预训练速度和性能。