Conventional NAS-based pruning algorithms aim to find the sub-network with
the best validation performance. However, validation performance does not
successfully represent test performance, i.e., potential performance. Also,
although fine-tuning the pruned network to restore the performance drop is an
inevitable process, few studies have handled this issue. This paper provides a
novel Ensemble Knowledge Guidance (EKG) to solve both problems at once. First,
we experimentally prove that the fluctuation of loss landscape can be an
effective metric to evaluate the potential performance. In order to search a
sub-network with the smoothest loss landscape at a low cost, we employ EKG as a
search reward. EKG utilized for the following search iteration is composed of
the ensemble knowledge of interim sub-networks, i.e., the by-products of the
sub-network evaluation. Next, we reuse EKG to provide a gentle and informative
guidance to the pruned network while fine-tuning the pruned network. Since EKG
is implemented as a memory bank in both phases, it requires a negligible cost.
For example, when pruning and training ResNet-50, just 315 GPU hours are
required to remove around 45.04% of FLOPS without any performance degradation,
which can operate even on a low-spec workstation. the implemented code is
available at this https URL.

本文提出一种基于集成知识引导的删减算法 (EKG)，通过评估不同子网络的损失波动情况搜索最佳的子网络并实现后续的剪枝及微调，仅需 315 GPU 小时即可在不降低性能的情况下移除约 45.04% 的 FLOPS。