Techniques in causal analysis of language models illuminate how linguistic
information is organized in LLMs. We use one such technique, AlterRep, a method
of counterfactual probing, to explore the internal structure of multilingual
models (mBERT and XLM-R). We train a linear classifier on a binary language
identity task, to classify tokens between Language X and Language Y. Applying a
counterfactual probing procedure, we use the classifier weights to project the
embeddings into the null space and push the resulting embeddings either in the
direction of Language X or Language Y. Then we evaluate on a masked language
modeling task. We find that, given a template in Language X, pushing towards
Language Y systematically increases the probability of Language Y words, above
and beyond a third-party control language. But it does not specifically push
the model towards translation-equivalent words in Language Y. Pushing towards
Language X (the same direction as the template) has a minimal effect, but
somewhat degrades these models. Overall, we take these results as further
evidence of the rich structure of massive multilingual language models, which
include both a language-specific and language-general component. And we show
that counterfactual probing can be fruitfully applied to multilingual models.

利用一个名为 AlterRep 的方法，运用因果分析技术对多语言模型的内部结构进行研究，发现大规模多语言模型中存在着既有语言特定的组件又有语言通用组件，并展示了反事实推理方法在多语言模型上的应用前景。

多语言模型中的语言身份反事实检测

Counterfactually Probing Language Identity in Multilingual Models

Multilingual BERT (mBERT) provides sentence representations for 104
languages, which are useful for many multi-lingual tasks. Previous work probed
the cross-linguality of mBERT using zero-shot transfer learning on
morphological and syntactic tasks. We instead focus on the semantic properties
of mBERT. We show that mBERT representations can be split into a
language-specific component and a language-neutral component, and that the
language-neutral component is sufficiently general in terms of modeling
semantics to allow high-accuracy word-alignment and sentence retrieval but is
not yet good enough for the more difficult task of MT quality estimation. Our
work presents interesting challenges which must be solved to build better
language-neutral representations, particularly for tasks requiring linguistic
transfer of semantics.

本文讨论了 Multilingual BERT (mBERT) 的语义属性，表明 mBERT 表征可以分为特定语言和语言中立两个部分，语言中立部分可以高准确度地对齐单词和检索句子，但尚不足以评估机器翻译质量。该研究揭示了构建更好的语言中立表征的挑战，特别是对于需要语义语言转移的任务。