Today a canonical approach to reduce the computation cost of Deep Neural
Networks (DNNs) is to pre-define an over-parameterized model before training to
guarantee the learning capacity, and then prune unimportant learning units
(filters and neurons) during training to improve model compactness. We argue it
is unnecessary to introduce redundancy at the beginning of the training but
then reduce redundancy for the ultimate inference model. In this paper, we
propose a Continuous Growth and Pruning (CGaP) scheme to minimize the
redundancy from the beginning. CGaP starts the training from a small network
seed, then expands the model continuously by reinforcing important learning
units, and finally prunes the network to obtain a compact and accurate model.
As the growth phase favors important learning units, CGaP provides a clear
learning purpose to the pruning phase. Experimental results on representative
datasets and DNN architectures demonstrate that CGaP outperforms previous
pruning-only approaches that deal with pre-defined structures. For VGG-19 on
CIFAR-100 and SVHN datasets, CGaP reduces the number of parameters by 78.9% and
85.8%, FLOPs by 53.2% and 74.2%, respectively; For ResNet-110 On CIFAR-10, CGaP
reduces 64.0% number of parameters and 63.3% FLOPs.

本文提出了 Continuous Growth and Pruning (CGaP) 方法，旨在在训练模型期间最小化冗余并从一开始便避免引入冗余，通过持续增长和修剪过程获取紧凑和准确的模型。根据实验结果，在代表性数据集和 DNN 结构上，CGaP 的效果优于以前的仅修剪预定义结构的方法，达到了参数减少 78.9％和 85.8％，FLOPs（浮点计算数）减少 53.2％和 74.2％。