Neural network pruning serves as a critical technique for enhancing the
efficiency of deep learning models. Unlike unstructured pruning, which only
sets specific parameters to zero, structured pruning eliminates entire
channels, thus yielding direct computational and storage benefits. However, the
diverse patterns for coupling parameters, such as residual connections and
group convolutions, the diverse deep learning frameworks, and the various time
stages at which pruning can be performed make existing pruning methods less
adaptable to different architectures, frameworks, and pruning criteria. To
address this, we introduce Structurally Prune Anything (SPA), a versatile
structured pruning framework that can prune neural networks with any
architecture, from any framework, and at any stage of training. SPA leverages a
standardized computational graph and ONNX representation to prune diverse
neural network architectures without the need for manual intervention. SPA
employs a group-level importance estimation method, which groups dependent
computational operators, estimates their importance, and prunes unimportant
coupled channels. This enables the transfer of various existing pruning
criteria into a structured group style. As a result, SPA supports pruning at
any time, either before training, after training with fine-tuning, or after
training without fine-tuning. In the context of the latter, we introduce
Optimal Brain SPA (OBSPA), an algorithm that achieves state-of-the-art pruning
results needing neither fine-tuning nor calibration data. In extensive
experiments, SPA shows competitive to state-of-the-art pruning performance
across various architectures, from popular frameworks, at different pruning
times.

结构化修剪技术 (SPA) 是一种适用于不同架构、框架和修剪标准的多功能架构修剪框架，通过使用标准化计算图和 ONNX 表示法来实现对神经网络的修剪，并通过分组级别的重要性估计方法来进行修剪。在对比实验中，SPA 在不同架构、流行框架和不同修剪时间上表现出与现有技术相当的修剪性能。OBSPA 是一种无需微调或校准数据即可实现最先进的修剪结果的算法。

结构剪枝：任何架构，任何框架，任何时间

Structurally Prune Anything: Any Architecture, Any Framework, Any Time

Soft filter pruning~(SFP) has emerged as an effective pruning technique for
allowing pruned filters to update and the opportunity for them to regrow to the
network. However, this pruning strategy applies training and pruning in an
alternative manner, which inevitably causes inconsistent representations
between the reconstructed network~(R-NN) at the training and the pruned
network~(P-NN) at the inference, resulting in performance degradation. In this
paper, we propose to mitigate this gap by learning consistent representation
for soft filter pruning, dubbed as CR-SFP. Specifically, for each training
step, CR-SFP optimizes the R-NN and P-NN simultaneously with different
distorted versions of the same training data, while forcing them to be
consistent by minimizing their posterior distribution via the bidirectional
KL-divergence loss. Meanwhile, the R-NN and P-NN share backbone parameters thus
only additional classifier parameters are introduced. After training, we can
export the P-NN for inference. CR-SFP is a simple yet effective training
framework to improve the accuracy of P-NN without introducing any additional
inference cost. It can also be combined with a variety of pruning criteria and
loss functions. Extensive experiments demonstrate our CR-SFP achieves
consistent improvements across various CNN architectures. Notably, on ImageNet,
our CR-SFP reduces more than 41.8\% FLOPs on ResNet18 with 69.2\% top-1
accuracy, improving SFP by 2.1\% under the same training settings. The code
will be publicly available on GitHub.

CR-SFP 是一种简单而有效的训练框架，通过学习一致的表示来改善软过滤器修剪后的模型准确性，同时减少网络的计算复杂度，适用于各种 CNN 架构。

CR-SFP：学习一致表示用于软滤波器裁剪

CR-SFP: Learning Consistent Representation for Soft Filter Pruning

Dynamic Sparse Training (DST) is a rapidly evolving area of research that
seeks to optimize the sparse initialization of a neural network by adapting its
topology during training. It has been shown that under specific conditions, DST
is able to outperform dense models. The key components of this framework are
the pruning and growing criteria, which are repeatedly applied during the
training process to adjust the network's sparse connectivity. While the growing
criterion's impact on DST performance is relatively well studied, the influence
of the pruning criterion remains overlooked. To address this issue, we design
and perform an extensive empirical analysis of various pruning criteria to
better understand their effect on the dynamics of DST solutions. Surprisingly,
we find that most of the studied methods yield similar results. The differences
become more significant in the low-density regime, where the best performance
is predominantly given by the simplest technique: magnitude-based pruning. The
code is provided at this https URL

本论文主要研究动态稀疏训练中修剪准则的影响，发现在低密度环境下，最优性能主要取决于最简单的技术：基于大小的修剪。