Large language models (LLMs) have demonstrated proficiency across various
natural language processing (NLP) tasks but often require additional training,
such as continual pre-training and supervised fine-tuning. However, the costs
associated with this, primarily due to their large parameter count, remain
high. This paper proposes leveraging \emph{sparsity} in pre-trained LLMs to
expedite this training process. By observing sparsity in activated neurons
during forward iterations, we identify the potential for computational
speed-ups by excluding inactive neurons. We address associated challenges by
extending existing neuron importance evaluation metrics and introducing a
ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that
Sparsity-Accelerated Training (SAT) achieves comparable or superior performance
to standard training while significantly accelerating the process.
Specifically, SAT achieves a $45\%$ throughput improvement in continual
pre-training and saves $38\%$ training time in supervised fine-tuning in
practice. It offers a simple, hardware-agnostic, and easily deployable
framework for additional LLM training. Our code is available at
this https URL

使用稀疏性加速预训练大型语言模型 (LLMs) 的训练过程，通过观察前向迭代中被激活的神经元的稀疏性，排除不活跃的神经元以提高计算速度，这一方法在实践中实现了与标准训练相媲美甚至更好的性能，持续预训练中吞吐量提升了 45%，在监督微调中节省了 38% 的训练时间，提供了一个简单、硬件无关且易于部署的额外 LLM 训练框架。