Multilingual Large Language Models (LLMs) achieve remarkable levels of
zero-shot cross-lingual transfer performance. We speculate that this is
predicated on their ability to align languages without explicit supervision
from parallel sentences. While representations of translationally equivalent
sentences in different languages are known to be similar after convergence,
however, it remains unclear how such cross-lingual alignment emerges during
pre-training of LLMs. Our study leverages intrinsic probing techniques, which
identify which subsets of neurons encode linguistic features, to correlate the
degree of cross-lingual neuron overlap with the zero-shot cross-lingual
transfer performance for a given model. In particular, we rely on checkpoints
of BLOOM, a multilingual autoregressive LLM, across different training steps
and model scales. We observe a high correlation between neuron overlap and
downstream performance, which supports our hypothesis on the conditions leading
to effective cross-lingual transfer. Interestingly, we also detect a
degradation of both implicit alignment and multilingual abilities in certain
phases of the pre-training process, providing new insights into the
multilingual pretraining dynamics.

多语言大型语言模型通过隐式对齐语言和神经元重叠达到零 - shot 跨语言转移性能，本研究使用内在探测技术通过检查点观察到神经元重叠和下游性能之间的高相关性，同时探测到预训练过程中隐式对齐和多语言能力的退化现象，为多语言预训练动态提供了新的见解。