We present a novel global compression framework for deep neural networks that
automatically analyzes each layer to identify the optimal per-layer compression
ratio, while simultaneously achieving the desired overall compression. Our
algorithm hinges on the idea of compressing each convolutional (or
fully-connected) layer by slicing its channels into multiple groups and
decomposing each group via low-rank decomposition. At the core of our algorithm
is the derivation of layer-wise error bounds from the Eckart Young Mirsky
theorem. We then leverage these bounds to frame the compression problem as an
optimization problem where we wish to minimize the maximum compression error
across layers and propose an efficient algorithm towards a solution. Our
experiments indicate that our method outperforms existing low-rank compression
approaches across a wide range of networks and data sets. We believe that our
results open up new avenues for future research into the global
performance-size trade-offs of modern neural networks. Our code is available at
this https URL

本文提出了一种新颖的全局压缩深度神经网络框架，通过使用低秩分解的方法在达到所需的整体压缩率的同时，对每层进行全面分析确定最佳的每层压缩比率，并利用等奇特展开定理和最优化算法来得到我们的解决方案。我们的实验表明，我们的方法在各种网络和数据集上优于现有的低秩压缩方法。

神经网络压缩：寻求最佳分层分解方法

Compressing Neural Networks: Towards Determining the Optimal Layer-wise  Decomposition

Deep Neural Network (DNN) is powerful but computationally expensive and
memory intensive, thus impeding its practical usage on resource-constrained
front-end devices. DNN pruning is an approach for deep model compression, which
aims at eliminating some parameters with tolerable performance degradation. In
this paper, we propose a novel momentum-SGD-based optimization method to reduce
the network complexity by on-the-fly pruning. Concretely, given a global
compression ratio, we categorize all the parameters into two parts at each
training iteration which are updated using different rules. In this way, we
gradually zero out the redundant parameters, as we update them using only the
ordinary weight decay but no gradients derived from the objective function. As
a departure from prior methods that require heavy human works to tune the
layer-wise sparsity ratios, prune by solving complicated non-differentiable
problems or finetune the model after pruning, our method is characterized by 1)
global compression that automatically finds the appropriate per-layer sparsity
ratios; 2) end-to-end training; 3) no need for a time-consuming re-training
process after pruning; and 4) superior capability to find better winning
tickets which have won the initialization lottery.

本文提出了一个基于动量优化（momentum-SGD-based optimization）的优化方法，用于压缩 DNN 模型，从而消除冗余参数，减少网络的复杂度。该方法采用全局压缩策略来自动找到层次的稀疏率，经过优化后能够无需耗费时间进行重训练的过程，具有更好的压缩效果和训练能力。