Large language models (LLMs) have been widely employed across various application domains, yet their black-box nature poses significant challenges to understanding how these models process input data internally to make predictions. In this paper, we introduce a precise and quantitative law that governs the learning of contextualized token embeddings through intermediate layers in pre-trained LLMs for next-token prediction. Our findings reveal that each layer contributes equally to enhancing prediction accuracy, from the lowest to the highest layer -- a universal phenomenon observed across a diverse array of open-source LLMs, built on architectures such as Transformer, RWKV, and Mamba. We demonstrate that this law offers new perspectives and insights to inform and guide practices in LLM development and applications, including model scaling, pre-training tasks, and information flow. Overall, our law enables more fine-grained approaches to the design, training, and interpretation of LLMs through scrutinizing their internal data processing mechanisms.

本研究解决了大型语言模型（LLMs）在理解内部输入数据处理过程中的黑箱问题。我们提出了一种定量法则，揭示每一层在提高下一标记预测的准确性方面均作出相等贡献。这一发现对LLM的设计、训练和应用具有重要影响，提供了新的指导思路。

大型语言模型中的下一标记预测法则