Network quantization is an essential procedure in deep learning for
development of efficient fixed-point inference models on mobile or edge
platforms. However, as datasets grow larger and privacy regulations become
stricter, data sharing for model compression gets more difficult and
restricted. In this paper, we consider data-free network quantization with
synthetic data. The synthetic data are generated from a generator, while no
data are used in training the generator and in quantization. To this end, we
propose data-free adversarial knowledge distillation, which minimizes the
maximum distance between the outputs of the teacher and the (quantized) student
for any adversarial samples from a generator. To generate adversarial samples
similar to the original data, we additionally propose matching statistics from
the batch normalization layers for generated data and the original data in the
teacher. Furthermore, we show the gain of producing diverse adversarial samples
by using multiple generators and multiple students. Our experiments show the
state-of-the-art data-free model compression and quantization results for
(wide) residual networks and MobileNet on SVHN, CIFAR-10, CIFAR-100, and
Tiny-ImageNet datasets. The accuracy losses compared to using the original
datasets are shown to be very minimal.

本文提出了一种无需数据的对抗知识蒸馏，通过合成数据来实现数据自由的网络量化，包括多个生成器和多个学生的多样化对抗样本，实现了 (wide) residual networks 和 MobileNet 在 SVHN、CIFAR-10、CIFAR-100 和 Tiny-ImageNet 数据集上最先进的数据自由模型压缩和量化效果。