Modern neural networks are often massively overparameterized leading to high
compute costs during training and at inference. One effective method to improve
both the compute and energy efficiency of neural networks while maintaining
good performance is structured pruning, where full network structures (e.g.
neurons or convolutional filters) that have limited impact on the model output
are removed. In this work, we propose Bayesian Model Reduction for Structured
pruning (BMRS), a fully end-to-end Bayesian method of structured pruning. BMRS
is based on two recent methods: Bayesian structured pruning with multiplicative
noise, and Bayesian model reduction (BMR), a method which allows efficient
comparison of Bayesian models under a change in prior. We present two
realizations of BMRS derived from different priors which yield different
structured pruning characteristics: 1) BMRS_N with the truncated log-normal
prior, which offers reliable compression rates and accuracy without the need
for tuning any thresholds and 2) BMRS_U with the truncated log-uniform prior
that can achieve more aggressive compression based on the boundaries of
truncation. Overall, we find that BMRS offers a theoretically grounded approach
to structured pruning of neural networks yielding both high compression rates
and accuracy. Experiments on multiple datasets and neural networks of varying
complexity showed that the two BMRS methods offer a competitive
performance-efficiency trade-off compared to other pruning methods.

基于贝叶斯模型缩减技术的结构化剪枝方法可以在保持高压缩率和准确性的同时，提高神经网络的计算和能量效率。