We assess how multilingual language models maintain a shared multilingual
representation space while still encoding language-sensitive information in
each language. Using XLM-R as a case study, we show that languages occupy
similar linear subspaces after mean-centering, evaluated based on causal
effects on language modeling performance and direct comparisons between
subspaces for 88 languages. The subspace means differ along language-sensitive
axes that are relatively stable throughout middle layers, and these axes encode
information such as token vocabularies. Shifting representations by language
means is sufficient to induce token predictions in different languages.
However, we also identify stable language-neutral axes that encode information
such as token positions and part-of-speech. We visualize representations
projected onto language-sensitive and language-neutral axes, identifying
language family and part-of-speech clusters, along with spirals, toruses, and
curves representing token position information. These results demonstrate that
multilingual language models encode information along orthogonal
language-sensitive and language-neutral axes, allowing the models to extract a
variety of features for downstream tasks and cross-lingual transfer learning.

本文使用 XLM-R 为案例研究，研究了多语言语言模型如何在维持共享多语言表示空间同时在每种语言中编码语言敏感信息，并证明了多语言语言模型沿着语言敏感和语言中性轴编码信息，使其能够提取下游任务和跨语言传递学习的各种特征。