Recently, large language models (LLMs) have achieved tremendous breakthroughs in the field of language processing, yet their mechanisms in processing multiple languages remain agnostic. Therefore, in this work we study the multilingual activation patterns of LLMs. By transforming the original Large Language Models (LLMs) into a Mixture of Experts (MoE) architecture, we analyze the expert activation patterns when processing various languages and demonstrate the connections of these activation patterns at the level of language families. We discover the existence of non-language-specific neurons as well as language-specific activation neurons. Further exploration even showcases that merely leveraging high-frequency activation neurons can accelerate inference while maintaining comparable performance. These findings shed light on the LLMs' multilingual processing mechanism, and are of significant importance in guiding the multilingual training and model pruning of LLMs.

通过将大型语言模型（LLMs）转换为专家混合（MoE）架构，分析其在处理各种语言时的专家激活模式，探索并发现非语言特定神经元和语言特定激活神经元的存在，并展示了仅利用高频激活神经元可以在保持相当性能的同时加速推理的可能性，这些发现为LLMs的多语言处理机制提供了启示，对指导LLMs的多语言训练和模型修剪具有重要意义。

解密巴别塔：探索大型语言模型中的多语言激活模式