In recent years, deep network pruning has attracted significant attention in
order to enable the rapid deployment of AI into small devices with computation
and memory constraints. Pruning is often achieved by dropping redundant
weights, neurons, or layers of a deep network while attempting to retain a
comparable test performance. Many deep pruning algorithms have been proposed
with impressive empirical success. However, existing approaches lack a
quantifiable measure to estimate the compressibility of a sub-network during
each pruning iteration and thus may under-prune or over-prune the model. In
this work, we propose PQ Index (PQI) to measure the potential compressibility
of deep neural networks and use this to develop a Sparsity-informed Adaptive
Pruning (SAP) algorithm. Our extensive experiments corroborate the hypothesis
that for a generic pruning procedure, PQI decreases first when a large model is
being effectively regularized and then increases when its compressibility
reaches a limit that appears to correspond to the beginning of underfitting.
Subsequently, PQI decreases again when the model collapse and significant
deterioration in the performance of the model start to occur. Additionally, our
experiments demonstrate that the proposed adaptive pruning algorithm with
proper choice of hyper-parameters is superior to the iterative pruning
algorithms such as the lottery ticket-based pruning methods, in terms of both
compression efficiency and robustness.

本文提出了 PQ Index (PQI) 作为衡量深度神经网络可压缩性的量化指标，并基于此开发了一种基于稀疏性信息的自适应裁剪算法 (SAP)，该算法在压缩效率和稳健性方面优于遗传算法等迭代型裁剪算法。