With the increasing popularity of deep learning on edge devices, compressing
large neural networks to meet the hardware requirements of resource-constrained
devices became a significant research direction. Numerous compression
methodologies are currently being used to reduce the memory sizes and energy
consumption of neural networks. Knowledge distillation (KD) is among such
methodologies and it functions by using data samples to transfer the knowledge
captured by a large model (teacher) to a smaller one(student). However, due to
various reasons, the original training data might not be accessible at the
compression stage. Therefore, data-free model compression is an ongoing
research problem that has been addressed by various works. In this paper, we
point out that catastrophic forgetting is a problem that can potentially be
observed in existing data-free distillation methods. Moreover, the sample
generation strategies in some of these methods could result in a mismatch
between the synthetic and real data distributions. To prevent such problems, we
propose a data-free KD framework that maintains a dynamic collection of
generated samples over time. Additionally, we add the constraint of matching
the real data distribution in sample generation strategies that target maximum
information gain. Our experiments demonstrate that we can improve the accuracy
of the student models obtained via KD when compared with state-of-the-art
approaches on the SVHN, Fashion MNIST and CIFAR100 datasets.

本论文提出了一种基于知识蒸馏的数据无需模型压缩框架，通过维护一个动态的生成样本集合并添加实际数据分布的限制，解决了现有数据无需压缩方法中可能存在的灾难性遗忘问题和人工数据分布不匹配问题。在 SVHN、Fashion MNIST 和 CIFAR100 数据集上与最先进的方法相比，表明可以提高通过知识蒸馏获得的学生模型的精度。

通过合成数据防止知识蒸馏中的灾难性遗忘和分布不匹配

Preventing Catastrophic Forgetting and Distribution Mismatch in  Knowledge Distillation via Synthetic Data

Recently, an extensive amount of research has been focused on compressing and
accelerating Deep Neural Networks (DNN). So far, high compression rate
algorithms require part of the training dataset for a low precision
calibration, or a fine-tuning process. However, this requirement is
unacceptable when the data is unavailable or contains sensitive information, as
in medical and biometric use-cases. We present three methods for generating
synthetic samples from trained models. Then, we demonstrate how these samples
can be used to calibrate and fine-tune quantized models without using any real
data in the process. Our best performing method has a negligible accuracy
degradation compared to the original training set. This method, which leverages
intrinsic batch normalization layers' statistics of the trained model, can be
used to evaluate data similarity. Our approach opens a path towards genuine
data-free model compression, alleviating the need for training data during
model deployment.

本论文提出三种从训练模型中生成合成样本的方法，用于压缩和微调量化模型，不需要真实数据支持，可以用于数据敏感情况，最佳方法与原始训练样本相比具有可忽略的准确度下降，此方法利用训练模型的内在批归一化层统计信息，可用于评估数据之间的相似性，为真正的数据无损模型压缩打开了道路，并在模型部署过程中减轻了对训练数据的需求。