Recently, large language models (LLMs) have achieved tremendous breakthroughs
in the field of language processing, yet their mechanisms in processing
multiple languages remain agnostic. Therefore, in this work we study the
multilingual activation patterns of LLMs. By transforming the original Large
Language Models (LLMs) into a Mixture of Experts (MoE) architecture, we analyze
the expert activation patterns when processing various languages and
demonstrate the connections of these activation patterns at the level of
language families. We discover the existence of non-language-specific neurons
as well as language-specific activation neurons. Further exploration even
showcases that merely leveraging high-frequency activation neurons can
accelerate inference while maintaining comparable performance. These findings
shed light on the LLMs' multilingual processing mechanism, and are of
significant importance in guiding the multilingual training and model pruning
of LLMs.

通过将大型语言模型（LLMs）转换为专家混合（MoE）架构，分析其在处理各种语言时的专家激活模式，探索并发现非语言特定神经元和语言特定激活神经元的存在，并展示了仅利用高频激活神经元可以在保持相当性能的同时加速推理的可能性，这些发现为 LLMs 的多语言处理机制提供了启示，对指导 LLMs 的多语言训练和模型修剪具有重要意义。