The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferability and discrepancies of scaling laws between Dense Models and Mixture of Experts (MoE) models. Through a combination of theoretical analysis and extensive experiments, including consistent loss scaling, optimal batch size and learning rate scaling, and resource allocation strategies scaling, our findings reveal that the power-law scaling framework also applies to MoE Models, indicating that the fundamental principles governing the scaling behavior of these models are preserved, even though the architecture differs. Additionally, MoE Models demonstrate superior generalization, resulting in lower testing losses with the same training compute budget compared to Dense Models. These findings indicate the scaling consistency and transfer generalization capabilities of MoE Models, providing new insights for optimizing MoE Model training and deployment strategies.

本研究探讨了密集模型与专家混合模型（MoE）在大语言模型（LLM）缩放法则中的可转移性与差异性。通过理论分析和广泛实验，研究发现MoE模型在相同训练资源下比密集模型展现出更优的泛化能力，这为优化MoE模型训练和部署策略提供了新见解。

模型架构间的缩放法则：大语言模型中密集模型与专家混合模型的比较分析