Existing pruning methods utilize the importance of each weight based on
specified criteria only when searching for a sparse structure but do not
utilize it during training. In this work, we propose a novel approach -
\textbf{M}agnitude \textbf{A}ttention-based Dynamic \textbf{P}runing (MAP)
method, which applies the importance of weights throughout both the forward and
backward paths to explore sparse model structures dynamically. Magnitude
attention is defined based on the magnitude of weights as continuous
real-valued numbers enabling a seamless transition from a redundant to an
effective sparse network by promoting efficient exploration. Additionally, the
attention mechanism ensures more effective updates for important layers within
the sparse network. In later stages of training, our approach shifts from
exploration to exploitation, exclusively updating the sparse model composed of
crucial weights based on the explored structure, resulting in pruned models
that not only achieve performance comparable to dense models but also
outperform previous pruning methods on CIFAR-10/100 and ImageNet.

本文提出了一种新颖的动态裁剪方法，称之为基于数量级注意力的动态裁剪 (MAP) 方法，该方法在前向和反向路径中均运用了权重的重要性来动态探索稀疏模型结构，可以在更高效的情况下实现密集模型的性能，并且在 CIFAR-10/100 和 ImageNet 数据集上的表现比以前的裁剪方法更好。