Language agnostic and semantic-language information isolation is an emerging
research direction for multilingual representations models. We explore this
problem from a novel angle of geometric algebra and semantic space. A simple
but highly effective method "Language Information Removal (LIR)" factors out
language identity information from semantic related components in multilingual
representations pre-trained on multi-monolingual data. A post-training and
model-agnostic method, LIR only uses simple linear operations, e.g. matrix
factorization and orthogonal projection. LIR reveals that for weak-alignment
multilingual systems, the principal components of semantic spaces primarily
encodes language identity information. We first evaluate the LIR on a
cross-lingual question answer retrieval task (LAReQA), which requires the
strong alignment for the multilingual embedding space. Experiment shows that
LIR is highly effectively on this task, yielding almost 100% relative
improvement in MAP for weak-alignment models. We then evaluate the LIR on
Amazon Reviews and XEVAL dataset, with the observation that removing language
information is able to improve the cross-lingual transfer performance.

采用几何代数和语义空间的新角度，提出 “语言信息移除” 方法，通过从多语种表示中分离语言身份信息，进一步研究语言无关和语义 - 语言信息分离的问题。该方法简单而高效，只使用简单的线性运算，并在多语种嵌入空间的弱对齐模型上实现了近乎 100％的显着性能提高。在 Amazon Reviews 和 XEVAL 数据集上进一步验证，显示该方法能够提高跨语言转移性能。