Neural network compression has recently received much attention due to the
computational requirements of modern deep models. In this work, our objective
is to transfer knowledge from a deep and accurate model to a smaller one. Our
contributions are threefold: (i) we propose an adversarial network compression
approach to train the small student network to mimic the large teacher, without
the need for labels during training; (ii) we introduce a regularization scheme
to prevent a trivially-strong discriminator without reducing the network
capacity and (iii) our approach generalizes on different teacher-student
models.
In an extensive evaluation on five standard datasets, we show that our
student has small accuracy drop, achieves better performance than other
knowledge transfer approaches and it surpasses the performance of the same
network trained with labels. In addition, we demonstrate state-of-the-art
results compared to other compression strategies.

本研究介绍了一种通过对抗网络压缩方法实现从深层精确的模型向更小的模型中转移知识的方法，该方法不需要使用标签进行训练，并在不同的师生模型上泛化；在五个固定的标准数据集上进行广泛的评估表明，该学生模型准确率略有下降，而且性能比其他知识传输方法更好，并且超越了同一网络在使用标签训练时的性能，并且对比其他压缩策略的表现也达到了现有的最佳水平。