Network pruning reduces the computation costs of an over-parameterized
network without performance damage. Prevailing pruning algorithms pre-define
the width and depth of the pruned networks, and then transfer parameters from
the unpruned network to pruned networks. To break the structure limitation of
the pruned networks, we propose to apply neural architecture search to search
directly for a network with flexible channel and layer sizes. The number of the
channels/layers is learned by minimizing the loss of the pruned networks. The
feature map of the pruned network is an aggregation of K feature map fragments
(generated by K networks of different sizes), which are sampled based on the
probability distribution.The loss can be back-propagated not only to the
network weights, but also to the parameterized distribution to explicitly tune
the size of the channels/layers. Specifically, we apply channel-wise
interpolation to keep the feature map with different channel sizes aligned in
the aggregation procedure. The maximum probability for the size in each
distribution serves as the width and depth of the pruned network, whose
parameters are learned by knowledge transfer, e.g., knowledge distillation,
from the original networks. Experiments on CIFAR-10, CIFAR-100 and ImageNet
demonstrate the effectiveness of our new perspective of network pruning
compared to traditional network pruning algorithms. Various searching and
knowledge transfer approaches are conducted to show the effectiveness of the
two components. Code is at: this https URL

本文提出了一种新的网络剪枝方法，通过神经结构搜索直接搜索灵活的通道和层大小， 最小化剪枝网络的失误实现通道和层大小的学习，提高了传统的网络剪枝方法， 并有效地区别了不同搜索和知识转移方法的效果。