With the increasing popularity of deep learning on edge devices, compressing
large neural networks to meet the hardware requirements of resource-constrained
devices became a significant research direction. Numerous compression
methodologies are currently being used to reduce the memory sizes and energy
consumption of neural networks. Knowledge distillation (KD) is among such
methodologies and it functions by using data samples to transfer the knowledge
captured by a large model (teacher) to a smaller one(student). However, due to
various reasons, the original training data might not be accessible at the
compression stage. Therefore, data-free model compression is an ongoing
research problem that has been addressed by various works. In this paper, we
point out that catastrophic forgetting is a problem that can potentially be
observed in existing data-free distillation methods. Moreover, the sample
generation strategies in some of these methods could result in a mismatch
between the synthetic and real data distributions. To prevent such problems, we
propose a data-free KD framework that maintains a dynamic collection of
generated samples over time. Additionally, we add the constraint of matching
the real data distribution in sample generation strategies that target maximum
information gain. Our experiments demonstrate that we can improve the accuracy
of the student models obtained via KD when compared with state-of-the-art
approaches on the SVHN, Fashion MNIST and CIFAR100 datasets.

本论文提出了一种基于知识蒸馏的数据无需模型压缩框架，通过维护一个动态的生成样本集合并添加实际数据分布的限制，解决了现有数据无需压缩方法中可能存在的灾难性遗忘问题和人工数据分布不匹配问题。在 SVHN、Fashion MNIST 和 CIFAR100 数据集上与最先进的方法相比，表明可以提高通过知识蒸馏获得的学生模型的精度。