Pre-trained language models (PLMs) contain vast amounts of factual knowledge,
but how the knowledge is stored in the parameters remains unclear. This paper
delves into the complex task of understanding how factual knowledge is stored
in multilingual PLMs, and introduces the Architecture-adapted Multilingual
Integrated Gradients method, which successfully localizes knowledge neurons
more precisely compared to current methods, and is more universal across
various architectures and languages. Moreover, we conduct an in-depth
exploration of knowledge neurons, leading to the following two important
discoveries: (1) The discovery of Language-Independent Knowledge Neurons, which
store factual knowledge in a form that transcends language. We design
cross-lingual knowledge editing experiments, demonstrating that the PLMs can
accomplish this task based on language-independent neurons; (2) The discovery
of Degenerate Knowledge Neurons, a novel type of neuron showing that different
knowledge neurons can store the same fact. Its property of functional overlap
endows the PLMs with a robust mastery of factual knowledge. We design
fact-checking experiments, proving that the degenerate knowledge neurons can
help the PLMs to detect wrong facts. Experiments corroborate these findings,
shedding light on the mechanisms of factual knowledge storage in multilingual
PLMs, and contribute valuable insights to the field. The source code will be
made publicly available for further research.

本研究揭示了多语种预训练语言模型中的事实知识是如何存储在参数中的，并引入了架构适应性多语种集成梯度方法，它相比现有方法更准确地定位知识神经元，并且在不同的架构和语言中更具普适性。此外，我们对知识神经元进行了深入探索，发现了两个重要发现：(1) 存储跨语言事实知识的语言无关知识神经元的发现；(2) 退化知识神经元的发现，它表明不同的知识神经元可以存储相同的事实。实验证明了这些发现，并为多语种预训练语言模型中事实知识存储的机制提供了宝贵的见解，源代码将公开供进一步研究使用。