We explore the impact of parameter sparsity on the scaling behavior of
Transformers trained on massive datasets (i.e., "foundation models"), in both
vision and language domains. In this setting, we identify the first scaling law
describing the relationship between weight sparsity, number of non-zero
parameters, and amount of training data, which we validate empirically across
model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to
characterize the "optimal sparsity", the sparsity level which yields the best
performance for a given effective model size and training budget. For a fixed
number of non-zero parameters, we identify that the optimal sparsity increases
with the amount of data used for training. We also extend our study to
different sparsity structures (such as the hardware-friendly n:m pattern) and
strategies (such as starting from a pretrained dense model). Our findings shed
light on the power and limitations of weight sparsity across various parameter
and computational settings, offering both theoretical understanding and
practical implications for leveraging sparsity towards computational efficiency
improvements.

我们探讨了参数稀疏性对在大规模数据集上训练的 Transformer（即 “基础模型”）的扩展行为的影响，涉及视觉和语言领域。我们首次确定了描述权重稀疏性、非零参数数量和训练数据量之间关系的扩展定律，并通过 ViT/JFT-4B 和 T5/C4 在模型和数据规模上进行了实证验证；这些结果使我们能够表征 “最佳稀疏度”，即对于给定的有效模型大小和训练预算，可以在其中获得最佳性能的稀疏度水平。我们发现，在非零参数数量固定时，最佳稀疏度随着用于训练的数据量增加而增加。我们还将研究扩展到了不同的稀疏结构（如硬件友好的 n:m 模式）和策略（如从预训练的稠密模型开始）。我们的发现揭示了在各种参数和计算设置中权重稀疏性的能力和局限性，为利用稀疏性提高计算效率提供了理论理解和实际意义。