Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We release the static embeddings and the continued pre-training code. Unlike most previous work, our continued pre-training approach does not require parallel text.

本文介绍了一种将静态嵌入和上下文嵌入相结合的方法，通过提取40种语言的静态嵌入并使用VecMap校准来改善多语言表示，与此同时，应用新颖的持续预训练方法来进一步提高XLM-R的表示精度，并在多个复杂语义任务中展现出积极的效果。

静态和上下文多语言嵌入的结合