Related works used indexes like CKA and variants of CCA to measure the
similarity of cross-lingual representations in multilingual language models. In
this paper, we argue that assumptions of CKA/CCA align poorly with one of the
motivating goals of cross-lingual learning analysis, i.e., explaining zero-shot
cross-lingual transfer. We highlight what valuable aspects of cross-lingual
similarity these indexes fail to capture and provide a motivating case study
\textit{demonstrating the problem empirically}. Then, we introduce
\textit{Average Neuron-Wise Correlation (ANC)} as a straightforward alternative
that is exempt from the difficulties of CKA/CCA and is good specifically in a
cross-lingual context. Finally, we use ANC to construct evidence that the
previously introduced ``first align, then predict'' pattern takes place not
only in masked language models (MLMs) but also in multilingual models with
\textit{causal language modeling} objectives (CLMs). Moreover, we show that the
pattern extends to the \textit{scaled versions} of the MLMs and CLMs (up to 85x
original mBERT).\footnote{Our code is publicly available at
https://github.com/TartuNLP/xsim}

该论文使用 Average Neuron-Wise Correlation (ANC) 作为跨语言上下文中跨语言表示相似度的替代指标，并提供了实证研究证明了先对齐再预测的跨语言学习问提存在于多语言模型中。