The success of multilingual pre-trained models is underpinned by their
ability to learn representations shared by multiple languages even in absence
of any explicit supervision. However, it remains unclear how these models learn
to generalise across languages. In this work, we conjecture that multilingual
pre-trained models can derive language-universal abstractions about grammar. In
particular, we investigate whether morphosyntactic information is encoded in
the same subset of neurons in different languages. We conduct the first
large-scale empirical study over 43 languages and 14 morphosyntactic categories
with a state-of-the-art neuron-level probe. Our findings show that the
cross-lingual overlap between neurons is significant, but its extent may vary
across categories and depends on language proximity and pre-training data size.

该研究采用神经元级别探测器，在包括 43 种语言的 14 个形态句法类别上进行了大规模的实证研究，并发现了多语言预训练模型的跨语言重叠是显著的，但其范围可能因语言接近性和预培训数据的大小而异，支持其能够对语法进行语言通用的抽象。