Activation sparsity refers to the existence of considerable
weakly-contributed elements among activation outputs. As a prevalent property
of the models using the ReLU activation function, it has been proven a
promising paradigm to boost model inference efficiency. Nevertheless, most
large language models (LLMs) adopt activation functions without intrinsic
activation sparsity (e.g., GELU and Swish). Some recent efforts have explored
introducing ReLU or its variants as the substitutive activation function to
help LLMs achieve activation sparsity and inference acceleration, but few can
simultaneously obtain high sparsity and comparable model performance. This
paper introduces an effective sparsification method named "ProSparse" to push
LLMs for higher activation sparsity without decreasing model performance.
Specifically, after substituting the activation function of LLMs with ReLU,
ProSparse adopts progressive sparsity regularization with a factor smoothly
increasing along sine curves in multiple stages. This can enhance activation
sparsity and alleviate performance degradation by avoiding radical shifts in
activation distribution. With ProSparse, we obtain high sparsity of 89.32% and
88.80% for LLaMA2-7B and LLaMA2-13B, respectively, achieving comparable
performance to their original Swish-activated versions. Our inference
acceleration experiments further demonstrate the practical acceleration brought
by higher activation sparsity.

本文介绍了一种名为 “ProSparse” 的有效稀疏化方法，通过将大型语言模型中的激活函数替换为 ReLU，并采用沿正弦曲线逐渐增加的因子的渐进稀疏正则化，实现了更高的激活稀疏性而不降低模型性能，从而提供了实际的推理加速。