Multilingualism in Large Language Models (LLMs) is an yet under-explored
area. In this paper, we conduct an in-depth analysis of the multilingual
capabilities of a family of a Large Language Model, examining its architecture,
activation patterns, and processing mechanisms across languages. We introduce
novel metrics to probe the model's multilingual behaviour at different layers
and shed light on the impact of architectural choices on multilingual
processing.
Our findings reveal different patterns of multilinugal processing in the
sublayers of Feed-Forward Networks of the models. Furthermore, we uncover the
phenomenon of "over-layerization" in certain model configurations, where
increasing layer depth without corresponding adjustments to other parameters
may degrade model performance. Through comparisons within and across languages,
we demonstrate the interplay between model architecture, layer depth, and
multilingual processing capabilities of LLMs trained on multiple languages.

多语言模型（LLMs）中的多语言能力和处理机制的体系结构、激活模式和处理方法的深入分析的研究结果表明模型的层次结构和深度对多语言处理能力有影响。