Multilingual pre-trained language models have demonstrated impressive
(zero-shot) cross-lingual transfer abilities, however, their performance is
hindered when the target language has distant typology from source languages or
when pre-training data is limited in size. In this paper, we propose XLM-P,
which contextually retrieves prompts as flexible guidance for encoding
instances conditionally. Our XLM-P enables (1) lightweight modeling of
language-invariant and language-specific knowledge across languages, and (2)
easy integration with other multilingual pre-training methods. On the tasks of
XTREME including text classification, sequence labeling, question answering,
and sentence retrieval, both base- and large-size language models pre-trained
with our proposed method exhibit consistent performance improvement.
Furthermore, it provides substantial advantages for low-resource languages in
unsupervised sentence retrieval and for target languages that differ greatly
from the source language in cross-lingual transfer.

本文提出了 XLM-P 模型，结合上下文获取灵活的编码，实现语言无关以及语言特定知识的轻量级建模，且可以与其他多语言预训练方法轻松集成，提供用于文本分类、序列标记、问答和句子检索的性能改进，对于低资源语言和跨语言传输具有实际优势。