A recent empirical observation of activation sparsity in MLP layers offers an
opportunity to drastically reduce computation costs for free. Despite several
works attributing it to training dynamics, the theoretical explanation of
activation sparsity's emergence is restricted to shallow networks, small
training steps well as modified training, even though the sparsity has been
found in deep models trained by vanilla protocols for large steps. To fill the
three gaps, we propose the notion of gradient sparsity as the source of
activation sparsity and a theoretical explanation based on it that explains
gradient sparsity and then activation sparsity as necessary steps to
adversarial robustness w.r.t. hidden features and parameters, which is
approximately the flatness of minima for well-learned models. The theory
applies to standardly trained LayerNorm-ed pure MLPs, and further to
Transformers or other architectures if noises are added to weights during
training. To eliminate other sources of flatness when arguing sparsities'
necessity, we discover the phenomenon of spectral concentration, i.e., the
ratio between the largest and the smallest non-zero singular values of weight
matrices is small. We utilize random matrix theory (RMT) as a powerful
theoretical tool to analyze stochastic gradient noises and discuss the
emergence of spectral concentration. With these insights, we propose two
plug-and-play modules for both training from scratch and sparsity finetuning,
as well as one radical modification that only applies to from-scratch training.
Another under-testing module for both sparsity and flatness is also immediate
from our theories. Validational experiments are conducted to verify our
explanation. Experiments for productivity demonstrate modifications'
improvement in sparsity, indicating further theoretical cost reduction in both
training and inference.

基于梯度稀疏性和随机矩阵理论的激活稀疏性，该研究解释了深度模型中激活稀疏性的理论机制以及其在对抗鲁棒性和性能方面的重要性，并提出了几种用于训练和稀疏调整的模块和修改的方法。