Large pretrained multilingual language models (ML-LMs) have shown remarkable
capabilities of zero-shot cross-lingual transfer, without direct cross-lingual
supervision. While these results are promising, follow-up works found that,
within the multilingual embedding spaces, there exists strong language identity
information which hinders the expression of linguistic factors shared across
languages. For semantic tasks like cross-lingual sentence retrieval, it is
desired to remove such language identity signals to fully leverage semantic
information. In this work, we provide a novel view of projecting away
language-specific factors from a multilingual embedding space. Specifically, we
discover that there exists a low-rank subspace that primarily encodes
information irrelevant to semantics (e.g., syntactic information). To identify
this subspace, we present a simple but effective unsupervised method based on
singular value decomposition with multiple monolingual corpora as input. Once
the subspace is found, we can directly project the original embeddings into the
null space to boost language agnosticism without finetuning. We systematically
evaluate our method on various tasks including the challenging
language-agnostic QA retrieval task. Empirical results show that applying our
method consistently leads to improvements over commonly used ML-LMs.

在这项工作中，我们提出了一种新的方法，通过从多语言嵌入空间中投影掉语言特定因素，以完全利用语义信息，并消除语言身份信号，从而在语义任务中取得更好的结果。

发现用于语言无关的多语言表示的低秩子空间

Discovering Low-rank Subspaces for Language-agnostic Multilingual  Representations

The size of the vocabulary is a central design choice in large pretrained
language models, with respect to both performance and memory requirements.
Typically, subword tokenization algorithms such as byte pair encoding and
WordPiece are used. In this work, we investigate the compatibility of
tokenizations for multilingual static and contextualized embedding spaces and
propose a measure that reflects the compatibility of tokenizations across
languages. Our goal is to prevent incompatible tokenizations, e.g., "wine"
(word-level) in English vs.\ "v i n" (character-level) in French, which make it
hard to learn good multilingual semantic representations. We show that our
compatibility measure allows the system designer to create vocabularies across
languages that are compatible -- a desideratum that so far has been neglected
in multilingual models.

本文就多语言的静态和情境化嵌入空间的分词兼容性问题提出了解决方案，可以帮助系统设计人员创建具有兼容性的跨语言词汇，从而提高多语言模型的性能。