While pretrained language models (PLMs) primarily serve as general purpose
text encoders that can be fine-tuned for a wide variety of downstream tasks,
recent work has shown that they can also be rewired to produce high-quality
word representations (i.e., static word embeddings) and yield good performance
in type-level lexical tasks. While existing work primarily focused on lexical
specialization of PLMs in monolingual and bilingual settings, in this work we
expose massively multilingual transformers (MMTs, e.g., mBERT or XLM-R) to
multilingual lexical knowledge at scale, leveraging BabelNet as the readily
available rich source of multilingual and cross-lingual type-level lexical
knowledge. Concretely, we leverage BabelNet's multilingual synsets to create
synonym pairs across $50$ languages and then subject the MMTs (mBERT and XLM-R)
to a lexical specialization procedure guided by a contrastive objective. We
show that such massively multilingual lexical specialization brings massive
gains in two standard cross-lingual lexical tasks, bilingual lexicon induction
and cross-lingual word similarity, as well as in cross-lingual sentence
retrieval. Crucially, we observe gains for languages unseen in specialization,
indicating that the multilingual lexical specialization enables generalization
to languages with no lexical constraints. In a series of subsequent controlled
experiments, we demonstrate that the pretraining quality of word
representations in the MMT for languages involved in specialization has a much
larger effect on performance than the linguistic diversity of the set of
constraints. Encouragingly, this suggests that lexical tasks involving
low-resource languages benefit the most from lexical knowledge of resource-rich
languages, generally much more available.

本文通过在 50 种语言中使用 BabelNet 这一丰富的跨语言单词知识库，将大规模多语言 Transformer（MMT）暴露于多语言词汇知识，并利用对比目标指导下的词汇专业化过程，大幅提高了双语词汇归纳、跨语言单词相似性和跨语言句子检索等交叉语言词汇任务的性能。同时，实验还表明，相对于语言多样性，MMT 的预训练质量对性能有更大的影响。