Pretrained multilingual language models (LMs) can be successfully transformed
into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional
fine-tuning or model distillation with parallel data. However, it remains
unclear how to best leverage them to represent sub-sentence lexical items
(i.e., words and phrases) in cross-lingual lexical tasks. In this work, we
probe SEs for the amount of cross-lingual lexical knowledge stored in their
parameters, and compare them against the original multilingual LMs. We also
devise a simple yet efficient method for exposing the cross-lingual lexical
knowledge by means of additional fine-tuning through inexpensive contrastive
learning that requires only a small amount of word translation pairs. Using
bilingual lexical induction (BLI), cross-lingual lexical semantic similarity,
and cross-lingual entity linking as lexical probing tasks, we report
substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI).
The results indicate that the SEs such as LaBSE can be 'rewired' into effective
cross-lingual lexical encoders via the contrastive learning procedure, and that
they contain more cross-lingual lexical knowledge than what 'meets the eye'
when they are used as off-the-shelf SEs. This way, we also provide an effective
tool for harnessing 'covert' multilingual lexical knowledge hidden in
multilingual sentence encoders.

本文将多语言预训练模型转化为多语言句子编码器，通过对句子编码器进行探索，采用基于对比学习的简单而高效的方法，揭示了句子编码器隐含的跨语言词汇知识，进而提供了一种利用跨语言词汇的工具。