Data-free Knowledge Distillation (DFKD) has attracted attention recently
thanks to its appealing capability of transferring knowledge from a teacher
network to a student network without using training data. The main idea is to
use a generator to synthesize data for training the student. As the generator
gets updated, the distribution of synthetic data will change. Such distribution
shift could be large if the generator and the student are trained
adversarially, causing the student to forget the knowledge it acquired at
previous steps. To alleviate this problem, we propose a simple yet effective
method called Momentum Adversarial Distillation (MAD) which maintains an
exponential moving average (EMA) copy of the generator and uses synthetic
samples from both the generator and the EMA generator to train the student.
Since the EMA generator can be considered as an ensemble of the generator's old
versions and often undergoes a smaller change in updates compared to the
generator, training on its synthetic samples can help the student recall the
past knowledge and prevent the student from adapting too quickly to new updates
of the generator. Our experiments on six benchmark datasets including big
datasets like ImageNet and Places365 demonstrate the superior performance of
MAD over competing methods for handling the large distribution shift problem.
Our method also compares favorably to existing DFKD methods and even achieves
state-of-the-art results in some cases.

本文提出一种名为 Momentum Adversarial Distillation (MAD) 的方法，使用演员生成器和 EMA（Exponential Moving Average）生成器的合成样本对学生进行培训，通过保持 EMA 生成器的过去版本来防止快速改变的学生，以解决 DFKD 中的分布偏移问题。实验证明，MAD 在许多基准数据集上胜过竞争方法，甚至在某些情况下实现了最新的成果。