Sparse neural networks are a key factor in developing resource-efficient
machine learning applications. We propose the novel and powerful sparse
learning method Adaptive Regularized Training (ART) to compress dense into
sparse networks. Instead of the commonly used binary mask during training to
reduce the number of model weights, we inherently shrink weights close to zero
in an iterative manner with increasing weight regularization. Our method
compresses the pre-trained model knowledge into the weights of highest
magnitude. Therefore, we introduce a novel regularization loss named
HyperSparse that exploits the highest weights while conserving the ability of
weight exploration. Extensive experiments on CIFAR and TinyImageNet show that
our method leads to notable performance gains compared to other sparsification
methods, especially in extremely high sparsity regimes up to 99.8 percent model
sparsity. Additional investigations provide new insights into the patterns that
are encoded in weights with high magnitudes.

提出了一种新颖且强大的稀疏学习方法：自适应正则化训练（ART）。通过逐步增加权重正则化，我们将稠密网络压缩为稀疏网络，并通过最高权重的方式压缩预训练模型的知识。在极高稀疏度情景下，我们的方法在 CIFAR 和 TinyImageNet 上比其他稀疏化方法具有显著的性能提升，并对高量级权重中编码的模式提供了新的认识。