Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the compute per example. While scaling typically involves increasing both, the precise interplay between these factors and their combined contribution to overall capacity remains not fully understood. We explore this relationship in the context of sparse Mixture-of-Expert models (MoEs), which allow scaling the number of parameters without proportionally increasing the FLOPs per example. We investigate how varying the sparsity level, i.e., the ratio of non-active to total parameters, affects model performance in terms of both pretraining and downstream performance. We find that under different constraints (e.g. parameter size and total training compute), there is an optimal level of sparsity that improves both training efficiency and model performance. These results provide a better understanding of the impact of sparsity in scaling laws for MoEs and complement existing works in this area, offering insights for designing more efficient architectures.

本研究解决了语言模型容量确切表现的理解差距，特别是参数数量与每个示例计算量之间的交互关系。通过探索稀疏混合专家模型的稀疏性水平对模型性能的影响，发现存在优化稀疏性水平能同时提高训练效率和模型性能的条件。这些发现为理解 MoE 的缩放规律中的稀疏性影响提供了新视角，有助于设计更高效的架构。

参数与FLOPs：混合专家语言模型最佳稀疏性的缩放规律