Multilingual pretrained language models (MPLMs) exhibit multilinguality and
are well suited for transfer across languages. Most MPLMs are trained in an
unsupervised fashion and the relationship between their objective and
multilinguality is unclear. More specifically, the question whether MPLM
representations are language-agnostic or they simply interleave well with
learned task prediction heads arises. In this work, we locate language-specific
information in MPLMs and identify its dimensionality and the layers where this
information occurs. We show that language-specific information is scattered
across many dimensions, which can be projected into a linear subspace. Our
study contributes to a better understanding of MPLM representations, going
beyond treating them as unanalyzable blobs of information.

本研究旨在了解多语言预训练语言模型（MPLMs）在不受监督学习方式下与其多语性之间的关系，指出 MPLMs 的表示是否是语言无关的或与学习任务预测头相互交错，同时，我们定位了 MPLM 中的语言特定信息，并确定其维度和发生这些信息的层次，以及该信息散布在许多维度中，并可投影为线性子空间。