Knowledge distillation has been widely used to produce portable and efficient
neural networks which can be well applied on edge devices for computer vision
tasks. However, almost all top-performing knowledge distillation methods need
to access the original training data, which usually has a huge size and is
often unavailable. To tackle this problem, we propose a novel data-free
approach in this paper, named Dual Discriminator Adversarial Distillation
(DDAD) to distill a neural network without any training data or meta-data. To
be specific, we use a generator to create samples through dual discriminator
adversarial distillation, which mimics the original training data. The
generator not only uses the pre-trained teacher's intrinsic statistics in
existing batch normalization layers but also obtains the maximum discrepancy
from the student model. Then the generated samples are used to train the
compact student network under the supervision of the teacher. The proposed
method obtains an efficient student network which closely approximates its
teacher network, despite using no original training data. Extensive experiments
are conducted to to demonstrate the effectiveness of the proposed approach on
CIFAR-10, CIFAR-100 and Caltech101 datasets for classification tasks. Moreover,
we extend our method to semantic segmentation tasks on several public datasets
such as CamVid and NYUv2. All experiments show that our method outperforms all
baselines for data-free knowledge distillation.

提出了一种名为 Dual Discriminator Adversarial Distillation (DDAD) 的新型无数据的知识蒸馏方法，通过生成样本，训练紧凑的学生网络，使其接近其教师网络，从而在计算机视觉任务中实现了高效的神经网络。

数据无关模型压缩的双判别器对抗蒸馏

Dual Discriminator Adversarial Distillation for Data-free Model  Compression

Knowledge distillation (KD) has proved to be an effective approach for deep
neural network compression, which learns a compact network (student) by
transferring the knowledge from a pre-trained, over-parameterized network
(teacher). In traditional KD, the transferred knowledge is usually obtained by
feeding training samples to the teacher network to obtain the class
probabilities. However, the original training dataset is not always available
due to storage costs or privacy issues. In this study, we propose a novel
data-free KD approach by modeling the intermediate feature space of the teacher
with a multivariate normal distribution and leveraging the soft targeted labels
generated by the distribution to synthesize pseudo samples as the transfer set.
Several student networks trained with these synthesized transfer sets present
competitive performance compared to the networks trained with the original
training set and other data-free KD approaches.

本研究提出了一个基于多元正态分布的数据自由知识蒸馏方法，该方法从过度参数化神经网络中模拟中间特征空间并利用软目标标签生成伪样本以作为转换集。使用这些合成的转移集训练多个学生网络，表现出与使用原始训练集和其他无数据知识蒸馏方法训练的网络相当的竞争性能。